Mainly the fact that there are a hell of a lot of people who travel with discounts they aren't entitled to. The fact that they are regularly finding people who are not paying the correct fare is surely justification for them doing it. It might be a minor inconvenience but I've never waited more than a few seconds, and if I had a Railcard, would have no problem showing it.
The current setup is due to DFT meddling and private operators squabbles over who should pay gate installation and staffing costs.