Yes, I have a huge problem with this. What those who disagree should do is take part and rationally prove that the motion is nonsense, not be shouting about "no-platforming". So let the racist or sexist speaker speak, then send them off with their tail behind their legs by way of quality debate.
One reason, indeed, for the fact that I am not represented directly by any party is that my view on many things (welfare state, provision of quality public services etc) is left of centre, but on free speech I am a complete purist - I agree with the freedom of speech elements of the First Amendment in the US and would support the same applying here.
Essentially I think anyone should be able to express any view they wish (views being separate from actions), however much I dislike it, but equally it is incumbent on people in society to be active in arguing against bad views like racism, sexism etc.