In that case, the will of the people would be clear. No problem with it.
Though you can use the same argument for literally any policy. If a party is elected to Government on a policy of say, zero immigration, then that is what that government should enact.
As for the harm, just imagine for a moment that the result had gone the other way... but then a party came into government a few years down the line who were elected on a policy of leaving the EU- no referendum, just unilateral withdrawal. Yes, that party would have a democratic mandate to do so. But the fact that there was a referendum on that issue, to me elevates it above a simple manifesto pledge, such that a reversal of that policy should be subject to a second referendum. Therefore in the real world, a move to rejoin should be subject to a referendum too.
In any case, any party putting a policy of rejoining (or holding a referendum on doing so) front and centre would be either brave or stupid* based on my admittedly subjective exposure to the opinions of others out in the real world. Even the remainers don't seem keen on going through the whole messy business again.
If it were to happen I think many on both sides (remain/leave, for avoidance of doubt) would become further disillusioned with politics in general, so turnouts would take a hit over the coming years. A lack of faith in democracy also tends to push those who don't abstain to the extremes.
*= come to think of it, given the way the Conservative campaign is going I wouldn't be surprised if Rishi announces that on his next awkward press conference at a rubber chicken factory... I wonder how many Labour supporters on here would "hold their nose and vote Tory"?