I think you're assuming as well that passenger numbers have decreased due to the new ticketing system, but they've been growing at over 2 million a year for the past two years. I suspect this is because the new system is effectively pushing people to the services they will pay for, e.g. if I would previously have been happy paying £100 for the first off-peak service, but if it's going to be £120 for that train then I'll instead wait a couple of hours and pay £70 for an advance, but I'll still travel. The side benefit of this is obviously that loadings are more spread out, so you're less likely to get people standing and increasing dwell times.
I don't think anyone's disagreeing that the LNER trial is achieving its intended purpose - as long as we recognise that the purpose is reducing subsidy, not simplifying fares. Demand is high enough on the railway that you can definitely get away with price-gouging travellers on the busiest trains, prioritising higher fare-payers, while lower farepayers have to adjust their lifestyle around the cheapest trains.
Where the debate lies is whether that's justified or not. For a service that ultimately is funded by the taxpayer (even if the trains aren't, the infrastructure is), which is meant to function as a public utility, just as the motorways are, to get from A to B across the country, any sort of favouritism or segregation between passengers, based on how much they're willing to pay, isn't exactly a progressive policy. The majority of the populace wouldn't accept government-sponsored schools/hospitals/motorways run for profit either with e.g. higher-bracket taxpayers getting waiting list priority.
My opinion is that some amount of optimising for cost-effectiveness is fully logical, it's simple budgeting strategy. But when you take it too far, looking at what exactly would reduce the cost of living the most, and drive the most modal shift, with the best value for taxpayer money, with no room for any leeway, you create societal habits that are hard to reverse. Public opinion 20 years from now can't be measured on a spreadsheet.
The long-term effect of such policies is almost always understated and underestimated - for example letting the long-distance railway network operate as a minimal-subsidy airline-style market will certainly over time create a status quo which pictures a high-quality Intercity/HSR network as a toy for the wealthy, while everyone else can take Lumo-style trains or drive. The result: you get a significant portion of your voterbase that just doesn't care at all about railway investment because it doesn't affect them, and car dependency for long-distance travel remains a thing for another few decades.
For example, most people in the UK rarely travel on British Airways, a fairly good quality airline, and therefore any BA developments or investments into Heathrow/City don't affect them at all, on the face of it at least. That's not because they wouldn't like a decent service offering, but because the budget airlines are the only affordable option for many. However, as it's been almost 30 years since the start of mainstream budget airlines (i.e. airline price-based segregation), people accept that they have to take the crappy option that's likely to be late, uncomfortable, and with few passenger rights, if they want to travel affordably abroad, because that's how it's been for so long, and it's too difficult to change it now. Before the railways reach this point, surely it's worth thinking about whether this is what's best for everyone.
That's not to mention that if GBR wants to fill empty seats on peak-time trains, releasing more advance fares is still an option, no need to remove the off-peak walk-up fare. It's not perfect - still no flexibility, but if we're not prepared to pay for a more affordable railway, while at the same time still wanting high utilisation
and no dynamic pricing, it's the only opton.
