Yes, sudden. According to the statistics at
https://commonslibrary.parliament.uk/research-briefings/sn06077/ net inward migration was well below 100K - typically around 50K for the 1990s up until 1997. Over the next 7 years it jumped to 268K in 2004, and has remained well above 200K in almost every year since then. Compared to the time it takes to build infrastructure, that's a pretty sudden jump - and one that, as far as we can tell - was largely not anticipated by the Government of the time.
So what? Yes, the Government arguably made mistakes in the mid 2000's. But in 2016 we were where we were and we had to respond to the situation at the time. And the situation in 2016 was that immigration levels had become unsustainably high, largely due to people taking advantage of FOM, and there was no way to fix the problem while remaining in the EU. Saying 'if only the Government had done something different in 2005' is not a policy option if the year is 2016.
True, but again, so what? Brexit did make a difference to the numbers of people coming from EU, which has helped the situation. The numbers of people coming from outside the EU is a different issue which will presumably require different solutions unrelated to Brexit. (And actually I'd argue that one good consequence of Brexit is that the reduction in FOM immigration means the Government now has more leeway to allow more immigration from outside the EU - which in many cases is made up of people who arguably have more need to come here - people escaping the repression in Hong Kong, for example).
I'm afraid that looks to me like the usual 'progressive'/'left-wing' tactic in arguments of trying to make up motives for your opponents instead of addressing people's actual concerns. Sure, I don't doubt that there are people who didn't want to be in the EU for other reasons, but the fact is that FOM lead to a vast increase in immigration into the UK - and that was a genuine concern for many people, and a reason in its own right why many people voted for Brexit.