I used the evening Uckfield trains to Norwood Junction for about 4 years, and have to say this doesn't ring true at all. They were horrendously overcrowded, but I don't recall any disorder issues. Nobody who wanted to go to Norwood Junction or East Croydon wanted to use such a busy service, or inconvenience Uckfield branch passengers, but we had no choice after previous timetable changes, especially while Thameslink was diverted away from London Bridge. Those trains were regularly the most overcrowded I've seen, but the main problem was them being 4 or 6 car, not oiks daring to mix with their betters. Hopefully the longer trains these days have helped?
I must be imagining the fare evaders, illegal "entertainers", drunken fights, etc. that have been ongoing for many years. I've worked either with or around (!) those trains for longer than your 4 years. It's a big deal, and if it's not very well managed indeed, it can result in quite a lot of complaints.
The trains are not long enough to hold Norwood Junction passengers (or the Sanderstead / Upper Warlingham folks, for that matter, and I'm sure they'd have you know that they'd hardly think of themselves as "oiks daring to mix with their betters"!).
This, in turn, means that they are often a warm, noisy, overcrowded, rowdy environment with little room for manoeuvre when somebody disagrees with someone else. They are also wholly unsuitable for Metro traffic, as they are designed as regional units. The Uckfield branch is actually about the shortest distance on which they are suitable for use.
I don't really see what the point with diverted Thameslink trains is. Up to now, these trains have very rarely ever called at Norwood Junction or any stations in that area, even before the diversion through Tulse Hill. As for East Croydon, it has other choices anyway, and a lot of venues are in easy enough reach of the East London Line, Blackfriars (for Thameslink) or other stops. Passengers from East Croydon also refuse to board the (often longer) trains around the same time.
The Delay Attribution and Train Planning people do take into account information about service loadings and passenger behaviour that they get sent by both passengers and staff. In fact, they're delighted to receive it, in general, as it gives them "real world" estimates to play with.
Lastly, you'll be interested to see that, at London Bridge, the Horsham and Brighton services immediately precede the Uckfield services on most of the evenings of the new timetable.
What does fare evasion or suspicion of it have to do with service improvement?
I think the idea is that if not enough people but a ticket then they cut the services. Obviously not great for those did biy a ticket but there will always be winners and losers would be a stock answer to that.
The ticket purchasing statistics drive station usage statistics on a lot of these routes with frequent stops. Independent surveys such as DfT checks will rarely be able to gauge the number of passengers "on the ground" who are short-faring by one or two stations, and if that's to the detriment of their local stops, so be it. There are few benefits to servicing a small railway station where many passengers see it as a convenience not worth paying for.
A surprising amount of the specification for which trains are needed is driven by
proven user requirements. The railways are not just run on the whims of the timetable planners.