I will actually say in his defence that he's left me alone in the last couple of months. He still stares at me, and is no doubt badmouthing me to colleagues when I've gone, but he's not attempted to hassle me or start studying my tickets and photo ID in great detail.
Perhaps he was spoken to, or just realised he isn't going to get anywhere with me. I'm not a fare evader, I don't upgrade to first class (except when it's declassified), don't put feet on seats or anything else that would give him any reason to do anything more than request my ticket and photocard.