ROC - have we created a single point of failure?

[modernrail]

I am not a railway professional (like Grayling - I don't even run the trains) and so forgive any incorrect terminology.

I have been caught up in the mess in West Yorkshire area twice, once when I think lightening hit the ROC and once when there was a power cut.

I was stick at Victoria the other night when a lineside fire meant the signal box (not sure whether that is an ROC?) had to be evacuated.

As these new ROC assets control vast areas, have we not created a single point of failure in such circumstances? Is the contingency planning and are the back-up power and other critical systems sufficient to acknowledge the critical importance of the asset?

My feeling is no from what I have experienced. A power cut in one place for instance should absolutely not be taking out the whole of West Yorkshire's trains.

 

[swt_passenger]

There was a thread about the recent Ashford failure:
https://www.railforums.co.uk/thread...4-for-freight-any-potential-solutions.175229/
Regards a ROC single point of failure, last significant thread I could find was in 2017, but there have been a few of them:
https://www.railforums.co.uk/threads/signal-box-failure-theoretical-question.144256/

 

[Bald Rick]

Only in the same way that any large control centre is a single point failure, eg Swanwick ATC.

 

[DarloRich]

Only in the same way that any large control centre is a single point failure, eg Swanwick ATC.

Agreed - any centralised control centre in any industry creates a similar situation.

 

[Tio Terry]

I am not a railway professional (like Grayling - I don't even run the trains) and so forgive any incorrect terminology.

I have been caught up in the mess in West Yorkshire area twice, once when I think lightening hit the ROC and once when there was a power cut.

I was stick at Victoria the other night when a lineside fire meant the signal box (not sure whether that is an ROC?) had to be evacuated.

As these new ROC assets control vast areas, have we not created a single point of failure in such circumstances? Is the contingency planning and are the back-up power and other critical systems sufficient to acknowledge the critical importance of the asset?

My feeling is no from what I have experienced. A power cut in one place for instance should absolutely not be taking out the whole of West Yorkshire's trains.

Whilst I have no personal knowledge of the incidents you mention I can talk in generalities on the design of ROC's.

Power comes from two separately sourced grid supplies and a further traction derived supply. The idea being that the probability of two grid failures is very unlikely but if that were to be the case then the traction supply could be utilised, if that also failed then there would not be many trains running anyway.

The building is designed to provide a two hour fire resistant operating floor which would allow the orderly shutdown of the railway in the event of a need to evacuate the building. All cabling within the building should be LSZH - Low Smoke Zero Halogen - to limit the spread of fire. All cable access routes through the building structure should be sealed to avoid the spread of smoke. The A/C is designed to stop the spread of smoke throughout the building and will normally shut down or change in to a different mode of operation in the event of a fire alarm activation. There's nothing particularly special in any of this, it's all pretty standard stuff for control centres.

It is French practice to design equipment rooms as Faraday Cages - such as those on HS1 - meaning that any lightning strike is most unlikely to cause damage to the contents of the building, this is not the practice, so far as I am aware, for Network Rail buildings.

Single point of failure? Yes, but then so was a single signal box which could paralyse an entire route. There are risks in both ways of operating a railway, the risks need to be identified and mitigated as much as possible.

 

[Ploughman]

Are there still local emergency panels?
Been in a few around Leeds area in the 90's

 

[MotCO]

I am also not a rail professional. If an ROC was disabled, how quickly can it be replicated elsewhere? Do other ROCs have any spare capacity or facilities to provide an alternative service?

Likewise, how quickly can a trackside box of gubbins be replicated? If one was disabled by accident or fire, how quickly can a new one be created? Are they bespoke, or full of circuit boards which need a 'simple' download of a computer program?

 

[Llanigraham]

I am also not a rail professional. If an ROC was disabled, how quickly can it be replicated elsewhere? Do other ROCs have any spare capacity or facilities to provide an alternative service?

Likewise, how quickly can a trackside box of gubbins be replicated? If one was disabled by accident or fire, how quickly can a new one be created? Are they bespoke, or full of circuit boards which need a 'simple' download of a computer program?

If a ROC is disabled then that is it. It cannot be replicated elsewhere. To do so would require the signallers at that Centre to be trained on both lines and there is not the capacity to do that.
Lineside cabinets are unique for each location, although there is much commonality of parts, and the time taken to repair would depend on how much damage has been caused andd where the S & T Teams are. They could be a couple of hours away in some areas.

 

[modernrail]

I have found this article from 2017 which should assist in the debate:

http://www.railtechnologymagazine.c...s-not-yet-adequately-assessed-by-network-rail

I have to say I find the line from NR about risks not being realised until these centres have been in operation for a while troubling. It should be possible to catch 99 per cent of the risks in advance with this sort of operation and any necessary contingency and back up should be in place for the opening of the ROC.

I would be interested to know where NR now thinks itself to be compared to this fairly flimsy 2017 benchmark.

 

[headshot119]

I am not a railway professional (like Grayling - I don't even run the trains) and so forgive any incorrect terminology.

I have been caught up in the mess in West Yorkshire area twice, once when I think lightening hit the ROC and once when there was a power cut.

I was stick at Victoria the other night when a lineside fire meant the signal box (not sure whether that is an ROC?) had to be evacuated.

As these new ROC assets control vast areas, have we not created a single point of failure in such circumstances? Is the contingency planning and are the back-up power and other critical systems sufficient to acknowledge the critical importance of the asset?

My feeling is no from what I have experienced. A power cut in one place for instance should absolutely not be taking out the whole of West Yorkshire's trains.

Lightening didn't hit the ROC it hit York IECC.

 

[nom de guerre]

I was stick at Victoria the other night when a lineside fire meant the signal box (not sure whether that is an ROC?) had to be evacuated.

The box evacuated was Ashford IECC. It’s not a ROC, although it was briefly classified as one before the scheme was revised from 14 locations to 12.

It opened in 1993, and after some subsequent expansion, has been controlling essentially the same area for around the last 20 years.

If an ROC was disabled, how quickly can it be replicated elsewhere? Do other ROCs have any spare capacity or facilities to provide an alternative service?

During the early stages of the ROC scheme, the stated aspiration was to allow any of the 14 (subsequently 12) ROCs to take over part/all of a sister location’s workstations during an emergency. This was quietly dropped once management realised how difficult and expensive this would be to implement.

As it stands, no ROC has the facilities or spare staffing capacity to allow it to cover another ROC’s “service”.

 

[MotCO]

During the early stages of the ROC scheme, the stated aspiration was to allow any of the 14 (subsequently 12) ROCs to take over part/all of a sister location’s workstations during an emergency. This was quietly dropped once management realised how difficult and expensive this would be to implement.

As it stands, no ROC has the facilities or spare staffing capacity to allow it to cover another ROC’s “service”.

If a ROC is disabled then that is it. It cannot be replicated elsewhere. To do so would require the signallers at that Centre to be trained on both lines and there is not the capacity to do that.

I find this distinctly worrying. So, if an ROC was destroyed by fire, it would take months to reinstate in an alternative location? And therefore no rail service? Or have I overstated this?

 

[Highlandspring]

I don’t think that business continuity/contingency plans for critical national infrastructure sites should be discussed on a public forum like this, except in the broadest possible terms.

 

[AlexNL]

Across the pond, centralisation of traffic control was finished two decades ago. Nowadays 13 ROCs control train traffic for the whole of the Netherlands (apart from yards & TMDs).

ProRail have invested in building a spare ROC which can be used in case of emergencies. Should disaster strike, then this location (near Utrecht) can take over any of the ROCs. It is manned by the original ROCs signallers, who are shuttled to the backup location.

The spare ROC is only used when the outage time is expected to be more than 4 hours, as it takes time to get the location online.

Source of information: https://www.prorail.nl/nieuws/verkeersleidingspost-den-haag-succesvol-uitgeweken

 

[MarkyT]

Across the pond, centralisation of traffic control was finished two decades ago. Nowadays 13 ROCs control train traffic for the whole of the Netherlands (apart from yards & TMDs).

ProRail have invested in building a spare ROC which can be used in case of emergencies. Should disaster strike, then this location (near Utrecht) can take over any of the ROCs. It is manned by the original ROCs signallers, who are shuttled to the backup location.

The spare ROC is only used when the outage time is expected to be more than 4 hours, as it takes time to get the location online.

Source of information: https://www.prorail.nl/nieuws/verkeersleidingspost-den-haag-succesvol-uitgeweken

The interesting thing about this is that the Netherlands is much smaller than the UK yet has about the same number of control centres today as proposed here. According to a rather dated study http://eprints.whiterose.ac.uk/2160/1/ITS_WP418_uploadable.pdf 'BR' has 16584 route km vs NS at 2798 route km, so about a six fold larger network.

 

[MarkyT]

In order that remote control of lineside signalling can be transferred more easily between control centres following an incident, it helps if the interlocking equipment is located at a number of different distributed sites, remote from the control workstations. While that remains true for a number of older relay interlockings re-controlled into ROCs in UK, for the last three decades processor-based interlocking practice has tended to co-locate all the interlockings at the control centre.

 

[Tio Terry]

In order that remote control of lineside signalling can be transferred more easily between control centres following an incident, it helps if the interlocking equipment is located at a number of different distributed sites, remote from the control workstations. While that remains true for a number of older relay interlockings re-controlled into ROCs in UK, for the last three decades processor-based interlocking practice has tended to co-locate all the interlockings at the control centre.

The original concept of ROC's - and I fully accept that things may have changed - was that they would not contain any interlocking. Just the workstations and their control and interface equipment. That would make it relatively easy to re-route data links from the interlocking to a different location in the event of a ROC being out of commission for any reason. I know that a number of new interlockings associated with Crossrail are remotely located and I suspect there are a large number of others. This would make it relatively easy to re-locate control but would also require signallers to be re-located also.

It's not just the signalling system that has to be considered, there's also the communications systems, phone and radio, to consider when arranging control from an alternative location.

 

[MarkyT]

I know that a number of new interlockings associated with Crossrail are remotely located and I suspect there are a large number of others. This would make it relatively easy to re-locate control but would also require signallers to be re-located also.

Having SSIs or similar modern equivalents located remotely from the signal box is nothing new either. The first instance I was aware of was in Cornwall where the short single line over St Pinnock and Largin viaducts on the otherwise double track main line was resignalled using SSI in the late 1980s (or thereabouts). The SSI cubicle was not located in the immediate vicinity of the controlling signalbox at Liskeard however; it was installed at Par instead, where the local maintenance technicians were based, so they could attend to it more easily, clearly. The SSI was working in conjuction with a small panel processor system at Liskeard rather than an IECC however. IECCs couldn't do such remote control for many years after that which dictated the architecture of many subsequent large schemes.

It's not just the signalling system that has to be considered, there's also the communications systems, phone and radio, to consider when arranging control from an alternative location.

Yes it is a mightily complex endeavor to design and build all of this switchover capability, and with the wide spacing of ROCs intended for UK, it would be necessary to transport a large signaller workforce a very long distance, just when the roads are in possible gridlock due to large scale cessation of rail services.

 

[Elecman]

Whilst I have no personal knowledge of the incidents you mention I can talk in generalities on the design of ROC's.

Power comes from two separately sourced grid supplies and a further traction derived supply. The idea being that the probability of two grid failures is very unlikely but if that were to be the case then the traction supply could be utilised, if that also failed then there would not be many trains running anyway.

The ROCs I deal with have 2 Independant DNO HV supplies and on site Standby Generators and external connections for anothe4 Standby generator. But no traction offtake

 

[Class 170101]

Don't let signallers burn the toast.

 

[DarloRich]

I don’t think that business continuity/contingency plans for critical national infrastructure sites should be discussed on a public forum like this, except in the broadest possible terms.

Quite right. Far to much info given away to people who don't need it or have a right to expect it.

All I will say is that people here seem to think that their brainwave is the first time the idea has been considered!

If a ROC burns down the impact would be the same as if the main ATC centre in the country burnt. Does that also worry you?

 

[edwin_m]

In order that remote control of lineside signalling can be transferred more easily between control centres following an incident, it helps if the interlocking equipment is located at a number of different distributed sites, remote from the control workstations. While that remains true for a number of older relay interlockings re-controlled into ROCs in UK, for the last three decades processor-based interlocking practice has tended to co-locate all the interlockings at the control centre.

However with SSIs and no doubt the more recent CBIs, if there is a spare set of hardware and backup copies of the data then it is relatively easy to set up a duplicate elsewhere. The communications to the trackside and the control centre can be routed over standard comms links. The rules relating to signalling mean it isn't as straightforward as might be expected, but it's going to be far easier than obtaining and wiring up thousands of relays and cables to replace the older generation of interlockings.

 

[MarkyT]

However with SSIs and no doubt the more recent CBIs, if there is a spare set of hardware and backup copies of the data then it is relatively easy to set up a duplicate elsewhere. The communications to the trackside and the control centre can be routed over standard comms links. The rules relating to signalling mean it isn't as straightforward as might be expected, but it's going to be far easier than obtaining and wiring up thousands of relays and cables to replace the older generation of interlockings.

I wasn't suggesting a new generation of relay interlockings Edwin! (Fairly) distributed remote processor-based interlockings near the track-side are the best compromise and are entirely feasible today as Tio Terry noted was the original ROC plan up thread. An orderly handover between control centres can then be orchestrated without interlocking shutdown/changeover, subject to all those other systems mentioned being similarly enabled, staff and workstations being available etc. A particular interlocking might even be left with simple auto working in 'through routes' during the changeover, just as many relay interlockings can be set in various degraded mode override control states when the primary remote control medium is disabled today.

 

[modernrail]

Quite right. Far to much info given away to people who don't need it or have a right to expect it.

All I will say is that people here seem to think that their brainwave is the first time the idea has been considered!

If a ROC burns down the impact would be the same as if the main ATC centre in the country burnt. Does that also worry you?

Well that depends upon the impact that would cause. If there is no ability to get an alternative up and running in a reasonable period, it would very much trouble me. There are also a whole host of scenarios other than a catastrophic fire.

I didn't ask for huge amounts of detail in the OP and so please do consider your tone towards those posting reasonable questions.

The point is also in the public domain already, as evidenced by the article I posted. If anybody feels they have posted information which should not be in the public domain, for contractual or other reasons, I would ask that they or the moderator removes it. The poster of this point may also want to suggest that NR removes the huge amount of information it appears to have put into to the public domain from the search I have just carried out, including locations!

I would consider that it is reasonable to ask questions about how how robust our railway is, especially at a time when it is not remotely robust. Lots of passenger and tax payer money has been invested in ROCs. It is important that the public has confidence in a railway that runs with the configuration specified by NR. Other of course have the right to disagree.

 

[NorthernSpirit]

Are there still local emergency panels?
Been in a few around Leeds area in the 90's

Greetland still has an emergency panel, its not far from where the signal box use to be.

 

[DanDaDriver]

If you think about it, some random event taking out an old wooden AB box in the middle of nowhere means that box is essentially switched out, becoming one big section.

The more boxes you have the more likely a random/accidental event is.

So therefore having one box diminishes the chance of a random catastrophe such as fire etc(not unlikely) but increases the chance of an organised atack (less likely)

 

[MarkyT]

Greetland still has an emergency panel, its not far from where the signal box use to be.

Remote relay interlockings operated via telemetry systems almost invariably contain a local emergency panel. They are very difficult to operate 'blind' as they don't have train describers or telephone and cab radio concentrators and usually you can't even see the railway as lineside equipment rooms rarely have windows. They were provided mainly for operation under instruction from the normal controlling panel box if the remote control telemetry link has completely failed or is disabled for engineering work. In more modern systems, similar levels of overall dependability are designed in by means of duplication of subsystems and diversity of communications routing.

 

[DarloRich]

Well that depends upon the impact that would cause. If there is no ability to get an alternative up and running in a reasonable period, it would very much trouble me. There are also a whole host of scenarios other than a catastrophic fire.

I didn't ask for huge amounts of detail in the OP and so please do consider your tone towards those posting reasonable questions.

The point is also in the public domain already, as evidenced by the article I posted. If anybody feels they have posted information which should not be in the public domain, for contractual or other reasons, I would ask that they or the moderator removes it. The poster of this point may also want to suggest that NR removes the huge amount of information it appears to have put into to the public domain from the search I have just carried out, including locations!

I would consider that it is reasonable to ask questions about how how robust our railway is, especially at a time when it is not remotely robust. Lots of passenger and tax payer money has been invested in ROCs. It is important that the public has confidence in a railway that runs with the configuration specified by NR. Other of course have the right to disagree.

I will be the one who decides what tone I use ( as long as it meets forum rules) thanks.

As I said you seem to think that the rather obvious scenarios you have dreamt up have not been considered. You are wrong. It is almost like there aren't whole teams of people thinking this stuff up!

It is clear from your opening post you want specific details about contingency plans rather than just to know there are contingency plans. As is pointed out above ROC are critical national assets and I do not feel it appropriate for people to share details of such plans with randoms.

If you are concerned write to nr and ask them what the contingency plans are. If you are a journo contact the duty press officer and ask them. They might even give you a tour!

 

[Tomnick]

If you think about it, some random event taking out an old wooden AB box in the middle of nowhere means that box is essentially switched out, becoming one big section.

It certainly wouldn’t become one big section immediately. Even if the box in question had the ability to switch out (relatively few have a block switch), the random event would presumably take the block circuits out too. It’d take a bit of work, presumably design and testing too, to get the adjacent boxes to be able to work to each other, although I suppose you could establish some form of degraded working between the two quite quickly.

 

[w1bbl3]

Well that depends upon the impact that would cause. If there is no ability to get an alternative up and running in a reasonable period, it would very much trouble me. There are also a whole host of scenarios other than a catastrophic fire.

Whilst I have no knowledge of NR's disaster recover / business continuity plans, I have professionally been involved in construction of numerous trading floors and (non rail) control centres together with DR locations for the same. Generally for the sort of business/service that has these type of centre they will have a detailed DR/BC plan covering total loss and all conceivable situations up to that including how long a period of business disruption is acceptable, for the former these will in board terms be one of two approaches 1. A full DR/BC location equipped ready for operation, 2. Contracts in place with equipment suppliers to provide and install the specialist kit needed in the control centre quickly and a contract in place with a temporary / modular building provider to provide the building. The DR plan would in this situation identify where the temporary building could be located including sizing and placement using the businesses existing estate.

The interesting and difficult bit usually isn't the plush building with people in it but the less plush equipment room or data centre where complex to install and difficult to source at sort notice equipment is needed. It is not uncommon for DR plans to call for "spares" to kept aside from normal operational spares for this purpose or where practical for the facility to be duplicated. This particularly common in banking/finance with spare DR data centres considered normal practice.

I'd expect NR's plan to look at lot like route two for ROC's and IECC's with a interruption period before an acceptable (but reduced/compromised) service is possible to be measured in days not months.