It must be remembered that public transport pricing policies are first and foremost a political decision.
It has been decided for many years that "public transport must pay its way". In order for that to happen, farebox income has to rise to meet the cost of infrastructure, staff, rolling stock, fuel and operations. This applies equally to buses as to trains, with the exception that road infastructure is shared with private motorists.
The way of doing this is twofold -
Pare the service back to the bare minimum which can continue to attract a steady (or preferably increasing) stream of revenue. This means running as-short-as-possible trains, dropping frequencies, limiting services over quieter times, etc.
The other way is to look at your captive markets and seek to increase fares to ensure the maximum return, while providing attractive discounts to entice new or occasional customers.
In the case of the rail network, additional complications arise. The highly political nature of the network makes changing service patterns, or fares, to meet demand difficult. You also have the issue of the high purchase costs of the stock, making it more appropriate to see if you can get any return during the slack periods (i.e., off peak), just to keep the leasing costs covered. The ORCATS system may also have undesirable effects on price promotions, but I would expect it's very hard to quantify.
Finally, keeping commuting prices high is very convenient if you want to avoid infrastructure expenditure. Pricing people off of the busiest trains has been discussed time and time again as a far cheaper way of matching demand and capacity.
I note again that this also applies to buses -- many operators charge higher fares in the peak, or offer a better value range of passes outside of the peak.
Although there's sound economics behind most of these premises, it is ultimately a decision that has come from Whitehall -- and not just this government, this has been going on for decades. So if you want to get it fixed, you need to start lobbying...