It could summed up by saying 'it's geography versus money' - you can make a railway line as flat and straight as you want *if* you have enough money to pay for the civil engineering to do it in hilly terrain (deep cuttings, tall embankments, long viaducts, long tunnels etc.).
Using the WCML as an example, the more favourable terrain 'down south' (combined with an easy sell to investors of lines linking the major cities of London, Birmingham, Manchester and Liverpool) meant that the London & Birmingham and the Grand Junction (Birmingham to the middle of the Liverpool and Manchester line) railways were built from the start as serious, fairly straight and flat, mainlines. North of Preston the potential traffic prospects were less and the terrain more difficult, so the available money from investors wouldn't stretch to building heroic, expensive, earthworks and structures etc. - they just exchanged curvature and gradients for lower construction costs.
--- old post above --- --- new post below ---
If you want an extreme example of the 'money versus terrain' problem, take a look at the building of the original
Canadian Pacific transcontinental route across Canada in the 1880s. A major reason why it was built was because the population British Columbia threatened to join the US federation instead of Canada unless they got a transport link to the eastern part of Canada.
To head off possible incursions from the US, they wanted to build it as far south as possible, but that meant tackling the Selkirk Mountains head-on (and Canada was a pretty impoverished country at the time). It ended up with very steep gradients (4% over Kicking Horse pass, plus using switchbacks to gain height), some massive bridges built entirely out of wood, and tight curves all over the place. A fabulously scenic railway that fufilled the brief but an expensive operating nightmare, the mountainous parts of which have been re-aligned and rebuilt at great cost over the subsequent years (and it's still a slow route). When they started building west from Calgary and east from Vancouver, they didn't even know if there *was* a usable pass through the mountains - almost at the last minute, Major A.B. Rogers found the pass that bears his name today - talk about heroic adventures...