I tend not to have an issue with the 'pay when challenged' brigade, so long as they do in fact pay. What tends to irk me a little is when those who clearly play the system - and therefore almost certainly get away with it on a regular basis - take offence on those occasions when you do get to them - and charge them the appropriate fare, as opposed to the cheapest possible ticket, to which they are no longer entitled.
An example; leaving a London terminus from an unbarriered platform, on a fully restricted service. Get to the very front coach mid way through the journey, punter requests ticket to small local shack way down the line (train does local stops towards the end of the trip), with a commonly used split. Punter is correctly advised that it will be a full fare Anytime for the full distance, weighing in at over £100. And boy is he displeased, several minutes arguing the toss including a full rundown of the terrible injustice being laid upon this poor innocent chap. The killer question; "Had I not approached you to ask for your ticket Sir, at what point would you have paid for your journey?", is met with a firm assertion that we would have come and found me - this question being asked about 45 minutes into the journey with him sat firmly in his seat in the front coach. Considering the potential for undesirable consequences, particularly if an interest were raised in his regular travelling habits, why draw such attention to yourself? You tried the dodge, many times it probably works a treat but alas tonight one is not lucky. Such is life; just pay and keep schtum!