This sounds a bit odd- surely there's something can be done with additional masts? It is, after all, essentially the same cellular communication protocol as used by mobile phones.
Hmmm, yes and no. Nym points to the amount of ferrous material in the ground, and I guess that could make a difference if you're going through a deep cutting with such rocks on either side, shielding the mobile terminal from the base stations.
This sort of technology works best if there's good line of sight to the base station, and in particularly rugged landscape that might be difficult without sticking more base stations in. Whether or not you do so depends on cost vs benefit really. If a base station in a particular location only gives you 200m in each direction, do you spend the many tens of thousands of pounds that one costs? If you already have a working cab radio system, and it's not a line where you require an in-cab signalling technology that's dependent on GSM-R, then why bother?
The mobile operators will do the same thing - they do tend to put base stations in where they're cost-effective; in particular, if there are two fairly straight sections of line coming up to each other at a curve, it's not uncommon for there to be a base station somewhere at that curve. The antennas on the base stations are fairly directional, so they'll be panned in such a way that they're pointing straight up or down the line. However, they'll still do the same thing - if there's a steep cutting or tunnel, and there's no chance of recouping the rather high cost of installing a new base station, then they simply won't do it.