Often many of the workers on farms would move around to work, as such if one farmer treated them badly they wouldn't go back, likewise if a country's system treated them badly they wouldn't go there.
The problem with "make the British do it" is that because of the need to move about with us having lost the ability to move within the EU freely is that any British don't it now have to only work in the UK, limiting what they can do.
The treatment of staff should be protected, with the rights of staff and responsibilities of employers clearly defined and enforced. It's something that the EU did by brining in rules to ensure that everyone is treated the same (such as the minimum of 20 days of leave for full time staff).
As to the impact on families, often those coming from the EU to do low paid jobs would do so when they were young (although like all things not always), with it being not unheard of for people to do so to earn enough to afford a home in their home country before then moving back to them carry on their life.
Likewise it's not unheard of people placing to do the above and then ending up staying (sometimes after then getting a job in what they are qualified in (for example as an architect) even though they started off doing low paid jobs. In staying they then settle down and start families.
It's also not unheard of young people sending money to their parents from their work elsewhere.
Yes for those who did have families (i.e. they are sending money to their wife to support her and the children) it would cause issues, however that is their choice. Now that may be that it's not really much of a choice, however I suspect that the numbers in that situation would likely be fairly limited in number. As such it could be argued that changing other rules rather than shutting down freedom of movement would be better.
Anyway, only the UK stoping freedom of movement isn't going to stop people do it and actually where the UK has 28 days as the minimum time off for full time workers those doing it could be better off than if they have to work in the next best country (earning wise) and only get 20 days of leave.
As such saying that we shouldn't have freedom of movement because people from other countries may be treated badly here without doing anything to stop them from being treated badly elsewhere isn't really helping them, is it?
As such I'd suggest it's a bit of a poor argument.
Likewise saying that low prices are important by only being able to provide then through badly treating your staff also isn't right.
However there is a middle way, have the countries within the free movement zone agree to minimum workers rights and ensure that there's ways to enforce then. Then over time ensure that the minimum rights improve so that there's less discrepancy between the best and worst.
Whilst that still may mean people moving to where jobs pay the most (like we see with people heading to London within the UK), the attractiveness of doing so reduces over time (not least due to the high cost of living in such places).
Given that London sea lots of people traveling there for work, with people commuting for long periods of time, potentially leaving their other half to raise the children in their own for much of the time, should we end freedom of movement to/from London within the UK or should we make it that conditions should be equal across the country so that fewer want to head to London for work? (It's a bit of an academic question as we already have equal, other than pay, conditions across the UK).