Farmers without any alternative jobs are getting poorer, a new report by Partnership for Economic Policy (PEP) has said.