Politically, I think that subsidy of freight operators is an easier "sell" to the electorate than subsidy of passenger operations. Many voters are private motorists who perceive that HGV traffic "gets in the way" on their car journeys; other voters object to the noise and pollution of HGVs in their neighbourhoods and their local town or city. Getting more freight on to the railways is a popular policy, even if it requires taxpayer subsidy.
It's a popular policy because noone knows how much it costs to get freight onto the railways - since the subsidy is hidden in the published cost of the passenger system.
Destroy the passenger system and the true subsidy cost will become obvious and the political consensus for the freight railway will melt away.
Right now the freight railway receives subsidies of hundreds of millions to billions of pounds a year with no comment, that would not occur when the passenger railway is stripped away.
A further issue is that mile for mile, the infrastructure operation and maintenance cost of a passenger railway is much higher than for a freight only line, because of the higher safety standards (e.g. TPWS, facing point locks), linespeeds, service intensity and track quality requirements.
I'm not sure how true this is, any more.
Even the US has given in and required the installation of Positive Train Control over huge portions of the US rail system, including lines with no real passenger services.
Given the huge operational and safety implications of derailments and other train control accidents, even freight trains are likely to require facing point blocks and the like in the future.
[Given that FPLs are now built into standard point machines, I doubt they are a significant cost in the modern era!]
On train speeds, higher train speeds impose higher maintenance requirements to some degree, sure, but a 10 tonne axle load multiple unit does far less track damage than a 20 tonne axle load freight train.
And whilst freight infrastructure might be cheaper unit for unit, you need a hell of a lot more of it.
For example, between Oxenholme Lake District and the Carlisle approach, if not for freight operations you'd neeed a grand total of six point machines - for the third platform and a set of crossovers at Penrith.
The freight railway imposes requirements several times that, especially with all the sidings.