ETCS Vulnerability to Radio Outage

[Belperpete]

There has been a radio comms issue today, reported in this thread:
Nationwide Signal Issue
i had assumed that a GSM failure would be fairly localised, say an individual radio mast, but the idea of a single failure taking out the whole national network is concerning.

While this has been reported as having nationwide impact on rail services, the routes most affected seem to be those with ETCS signalling. Significantly services through the Thameslink core, which would have propagated impact to other routes. All services on the Cambrian were also cancelled, and that would likely have impacted North Wales services sharing the same diagrams. The Elizabeth line was also reported as badly affected, probably because that uses ETCS to Heathrow.

Currently, very few lines are fitted with ETCS, but of course it is proposed to roll it out nationwide. If such a fault were to happen in the future, once ETCS has been significantly rolled out, could it lead to a sudden national shutdown? This does seem to be a very big vulnerability.

 

[yorksrob]

Seems to be the risk with any centralised control.

 

[birchesgreen]

Currently, very few lines are fitted with ETCS, but of course it is proposed to roll it out nationwide. If such a fault were to happen in the future, once ETCS has been significantly rolled out, could it lead to a sudden national shutdown? This does seem to be a very big vulnerability.

I bet there are some interested folk in Moscow, Tehran, Beijing et cetera looking at this.

 

[Belperpete]

Seems to be the risk with any centralised control.

Agreed, but even with ETCS the whole network won't be centrally controlled from one place. The Cambrian isn't controlled from the same place as the Thameslink core, yet one failure in Stoke seems to have taken them both out.

 

[yorksrob]

Agreed, but even with ETCS the whole network won't be centrally controlled from one place. The Cambrian isn't controlled from the same place as the Thameslink core, yet one failure in Stoke seems to have taken them both out.

Food for thought, certainly !

 

[Phillipimo]

All services on the Cambrian were also cancelled

RealTimeTrains suggests there were a couple of cancellations at Aberystwyth and delays on all lines but everything from Pwllheli ran this morning.

 

[westcoaster]

As far as I'm aware everything still worked ok, just trains were unable to set up.

Not sure where the OP got mass disruption through the TL core came from, I looked from 0600 onwards only a couple of cancellations seen.
Listed as planning error and points failure.

 

[edwin_m]

i understand that GSM-R will need to be upgraded to a new system, partly due to obsolescence (being based on 2G GSM) but also because it doesn't have the bandwidth to support widespread ETCS operation. That should create an opportunity to look closely at potential single point failures and security loopholes.

 

[Phillipimo]

As far as I'm aware everything still worked ok, just trains were unable to set up.

Not sure where the OP got mass disruption through the TL core came from, I looked from 0600 onwards only a couple of cancellations seen.
Listed as planning error and points failure.

Certainly plenty of delays and a few cancellations through the Thameslink core this morning but looking on Recent Train Times it was actually worse yesterday!

 

[Belperpete]

RealTimeTrains suggests there were a couple of cancellations at Aberystwyth and delays on all lines but everything from Pwllheli ran this morning.

I was basing it on what TfW were posting on Journey Check at the time, that all services Mach - Pwllheli and Shrewsbury -Aberystwyth were suspended, with road replacement being sourced.

They probably had no idea how long the problem was going to last. It can be difficult to estimate how long it will take to trace and fix such a fault, it is not something pre planned that you can give a timescale for. They were probably right to assume the worst, rather than just sit on their hands and hope it was going to be fixed quickly.

It sounds like the issue was resolved quite quickly, within 15 minutes according to some reports. But if it can cause such disruption with only a handful of lines operating with ETCS, I dread to think of the impact had the GWML, WCML AND ECML all been fitted.

 

[Haywain]

But if it can cause such disruption

It wasn't purely about lines using ETCS, but disruption becomes more widespread because one cancellation can always mean several more because the stock and crew are then not in the right places for their next workings, and other crew may be using those cancelled trains to get to their own next workings. On top of this, if the problem this morning was at a time when a good number of trains were entering service for the day, the impact is likely to be worse.

 

[nwales58]

I was basing it on what TfW were posting on Journey Check at the time, that all services Mach - Pwllheli and Shrewsbury -Aberystwyth were suspended, with road replacement being sourced.

They probably had no idea how long the problem was going to last. ... They were probably right to assume the worst

First couple of services arrived at Pwllheli around right time, so I was wondering why people thought the Cambrian was closed.

Looking at RTT, first one left Mach 20 minutes late but that is easily made up. First southbound left 20 late, 10 late at Machynlleth. Looks like a better than average day in the current era of not enough units to run the timetable.

The point is that although ETCS potentially adds more electronic points of failure the most common ones can be engineered around. The GSM-R registration failures clearly were not a *single* point of failure as only some attempts failed, a SPoF would have stopped the entire network.

After the initial pain, in the 13-14 years it has been in use on the Cambrian the only event I can remember bringing everything to a halt was the evaporating temporary speed restrictions mess, which was transfer from another system rather than inherent in ETCS itself.

 

[renegademaster]

Even if you distributed out the hardware , you'd still have the same issues because the servers would be having the same software update, with the same bug. Sounds more like a redundancy problem than a centralisation problem

 

[Harpo]

One instance of this type of failure in well over a decade and resolved in a few hours and we have a tabloid-esque ‘vulnerability’?

As always, resilience costs and this is similar to winter resilience etc.. How much does anyone want to spend on a smallish 1-in-13 year risk? It’s money better far spent on running better train services for the other 12.97-ish years.

Meanwhile, hopefully the logic of doing a weeknight system upgrade will get investigated.

 

[Belperpete]

One instance of this type of failure in well over a decade and resolved in a few hours and we have a tabloid-esque ‘vulnerability’?

As always, resilience costs and this is similar to winter resilience etc.. How much does anyone want to spend on a smallish 1-in-13 year risk? It’s money better far spent on running better train services for the other 12.97-ish years.

The point is that currently only a miniscule proportion of the network is controlled by ETCS, yet the impact was significant. What would the impact have been like if, as intended, the majority of the network were controlled by ETCS? And would that be acceptable every 1 in 13 years?

And were we unlucky that it took 15 minutes to resolve? Or were we lucky that it didn't take significantly longer?

Agreed that resilience costs. BR and NR have spent significant sums of money to ensure diverse routing of signalling data links, sometimes just to ensure that just a mile or two of line doesn't lose signalling. So the idea that a single failure could potentially take out the signalling over the whole country (once ETCS fitted) is very concerning.