Alignment is often a factor. The fast lines in most cases are the original lines so have the best and most straight alignment, but when the slow lines were added later they were less straight and curve away at points where the lines need to be seperated. A good example is stations, on a straight track the fast lines will be straight, but the slow lines will have curves to allow them to move away from the fast lines to provide room for the island platform/s and then return alongside them. Tunnels too, the slow lines will often curve away to pass through a tunnel, whilst the fast lines will be straighter.
At the end of the day the majority of traffic on the slow lines will be travelling slower. With the exception of Mail (which is rare these days), all freight is restricted to 75mph or less, most 60mph or less. Many suburban trains running on the slow lines are also restricted to 75mph. Due to the short distances between stations it's usually not worth having them going faster than that. You put a lot of energy in to accelerate them to higher speed, but even with regenerative braking (which is a very new thing) you loose most of it and over short distances you get very little advantage. Even where fast trains run on the slow lines, such as diversions, the lower speed doesn't make a massive difference, and if they were running faster they would just catch up with the stopping trains anyway. As had also been said, maintenance costs are higher where speeds are higher.