I don't get that. First of all, it's highly unlikely that the Conservatives would fight the election solely on that issue, and the other parties would fight on other issues which are important and the Conservatives wouldn't want to be saying they weren't interested in those when other parties, and many voters, considered they matter. Elections are always about more than one issue, except occasionally in a particular constituency when there's an issue that's sufficiently important locally to override everything else.
Secondly, I think if the Conservatives did fight on just "Stop the boats..." etc., and got trounced, most people would understand that there were many other reasons why they had lost going back over the past thirteen years, and it wouldn't be logical to conclude that immigration isn't such a big issue, partly because....
Thirdly, it is a big issue. It played a part in many people's decision to vote to leave the EU. They may have confused legal and illegal immigration, but they were against it, and still are. If they didn't like Poles taking over their town - as it seemed to them - they're unlikely to look any more favourably on Afghans or people from sub-Saharan Africa. In any case, there are issues about our increasing population and the additional homes and other facilities that requires.
Finally, if your scenario came to pass, how would that cause the next government to "begin to allow us our freedom of movement with the EU"? That matter is principally for the EU to decide, and they would want something from us in return - maybe quite a lot.