ROC - have we created a single point of failure?

[edwin_m]

I wasn't suggesting a new generation of relay interlockings Edwin! (Fairly) distributed remote processor-based interlockings near the track-side are the best compromise and are entirely feasible today as Tio Terry noted was the original ROC plan up thread. An orderly handover between control centres can then be orchestrated without interlocking shutdown/changeover, subject to all those other systems mentioned being similarly enabled, staff and workstations being available etc. A particular interlocking might even be left with simple auto working in 'through routes' during the changeover, just as many relay interlockings can be set in various degraded mode override control states when the primary remote control medium is disabled today.

Another factor to consider is that a central ROC is likely to have technicians on site, who can attend to faults more quickly than for equipment located remotely. This may somewhat worsen the consequences of losing the central site but ought to speed up the response to various lesser faults.

If you think about it, some random event taking out an old wooden AB box in the middle of nowhere means that box is essentially switched out, becoming one big section.

The more boxes you have the more likely a random/accidental event is.

So therefore having one box diminishes the chance of a random catastrophe such as fire etc(not unlikely) but increases the chance of an organised atack (less likely)

Not just equipment, staff too. A traditional AB box is likely to be single-manned and if someone calls in sick or has transport problems etc (or even gets locked in the toilet!) the box may be left empty. Several disruptions from these causes have been reported on this forum, including a possible one today. As mentioned boxes can't always be switched out, for example if controlling a level crossing, and it can't be done remotely. Even some of the smaller power boxes might struggle if say one out of three rostered staff wasn't available, but the bigger the establishment the easier it is to cover if someone is unexpectedly absent.

 

[modernrail]

I will be the one who decides what tone I use ( as long as it meets forum rules) thanks.

As I said you seem to think that the rather obvious scenarios you have dreamt up have not been considered. You are wrong. It is almost like there aren't whole teams of people thinking this stuff up!

It is clear from your opening post you want specific details about contingency plans rather than just to know there are contingency plans. As is pointed out above ROC are critical national assets and I do not feel it appropriate for people to share details of such plans with randoms.

If you are concerned write to nr and ask them what the contingency plans are. If you are a journo contact the duty press officer and ask them. They might even give you a tour!

As I asked the question, I will decide the level of detail I was seeking thanks

To clarify though, the point of particular interest for me is whether the ROC approach vs what I suppose was a distributed system is more likely to cause problems across a whole area versus delays to services on discreet routes, how this washes out in overall downtime numbers and how this is risk managed in the design process and in operation. The reason for my interest is twofold. First, I have been caught up in a number of of delays recently where there was no workaround by going the long way round, because the whole area is down. Secondly, I am involved in the development of distributed energy systems and resilience is a hot topic in that discussion. I often tell people to look at the rail sector when carrying out risk analysis as there should be useful lessons to learn.

Do NR split down the reasons for delays caused by NR infrastructure so you can see what type of asset has been involved in the delay? I would be interested to see the trend over the last few years if that breakdown is available but I haven't managed to find it yet.

 

[noddingdonkey]

As I said you seem to think that the rather obvious scenarios you have dreamt up have not been considered. You are wrong. It is almost like there aren't whole teams of people thinking this stuff up!

It may have been considered, but has not been adequately addressed, as the farce when York IECC got hit by lightning a few months ago demonstrated. I was stuck on Huddersfield platform 1 trying to get the Manchester, looking at the PSB which had just closed and would have allowed at least a Huddersfield-Manchester shuttle operation to have continued.

 

[MarkyT]

It may have been considered, but has not been adequately addressed, as the farce when York IECC got hit by lightning a few months ago demonstrated. I was stuck on Huddersfield platform 1 trying to get the Manchester, looking at the PSB which had just closed and would have allowed at least a Huddersfield-Manchester shuttle operation to have continued.

That was the old IECC, now defunct. Control has since passed to a new building, hopefully better protected.

 

[Bald Rick]

Do NR split down the reasons for delays caused by NR infrastructure so you can see what type of asset has been involved in the delay? I would be interested to see the trend over the last few years if that breakdown is available but I haven't managed to find it yet.

Some data is out there, but probably not at the level you want.

Delay incidents caused by the failure of a whole control centre or signal box are really rather rare, and becoming less so. In my experience they are far less prevalent with ROCs than the IECCs, Signalling Centres, Power Signal Boxes and plain vanilla signal boxes that preceded them. I can remember numerous incidents involving the older type, ranging from complete power failures, fire alarms, security alerts, water ingress (external and, ahem, internal), staff absences, and others. Yet in the 10 years or so that ROCs have been up and running, the number of service affecting shut down events that I can remember is one-hand-countable.

It may have been considered, but has not been adequately addressed, as the farce when York IECC got hit by lightning a few months ago demonstrated. I was stuck on Huddersfield platform 1 trying to get the Manchester, looking at the PSB which had just closed and would have allowed at least a Huddersfield-Manchester shuttle operation to have continued.

This is the luck of the draw. Had lightning struck Huddersfield PSB when it was open, you’d have been in the same boat, but Leeds and York would have been fine

Whilst ROCs do cover / will eventually cover large areas, and in the event of total failure will cause significant network disruption, it not just ROCs. The same is also true of plenty of other signalling control locations. Some of them are rather less obvious than you might think.

 

[Llanigraham]

It may have been considered, but has not been adequately addressed, as the farce when York IECC got hit by lightning a few months ago demonstrated. I was stuck on Huddersfield platform 1 trying to get the Manchester, looking at the PSB which had just closed and would have allowed at least a Huddersfield-Manchester shuttle operation to have continued.

I can think of atleast 2 instances where lightning strikes at mechanical signal boxes has resulted in line closures over several hours, proving that it isn't just ROC's that may be affected.

 

[LAX54]

Even lower grade boxes can scupper the service, if you take Stowmarket as an example, if that has to close, then there is nothing from Ipswich to Bury, and Ipswich to Diss, GA will run a bus service between these two points extending the journey time a great deal.
Brundall closed. nothing Norwich to Yarmouth and Lowestoft, so just a small box can have as much effect as a larger box.

PSB's / IECC's /ROCs all run short from time to time, but the staff within the boxes, shorten / forgo breaks in order to keep the service running, not that , that is ever appreciated, passengers just see a delay and they are 10 mins late !

 

[duffield]

My understanding (from publicly available information, and as mentioned in a previous post) is that effectively resilience has been reduced to control costs, i.e. that it was 'too difficult' (for which I would read mostly 'too expensive', but also in some case ineffective) to provide the originally planned level of resilience (operations transferred fairly swiftly to another ROC).

Note I'm not saying this was a good or bad decision; the level of resilience of individual ROCs may be such that it would be extravagant to provide the inter-ROC resilience originally envisaged. Only time will tell if this was the right call. For example, a ROC incident might have the same impact as a full-on TOC strike for a couple of weeks, but if that happens once in 20 years it might be reasonable to accept that risk.

However I do find it disappointing that it proved unfeasible to provide that level of resilience at a reasonable cost given that it was promoted (as far as I remember) as one of the main benefits of the the ROCs project in the first place.

 

[noddingdonkey]

This is the luck of the draw. Had lightning struck Huddersfield PSB when it was open, you’d have been in the same boat, but Leeds and York would have been fine

I can think of atleast 2 instances where lightning strikes at mechanical signal boxes has resulted in line closures over several hours, proving that it isn't just ROC's that may be affected.

Not sure that argument holds water. If a PSB or mechanical box was hit it's not the entire north of England where the job is stopped, other routes would remain open allowing diversions and only partial bustitution to a station which has a service rathe than the whole route. Inconvenient for sure, but less of an impact than if all of the eggs are in one basket.

 

[Bald Rick]

I can think of atleast 2 instances where lightning strikes at mechanical signal boxes has resulted in line closures over several hours, proving that it isn't just ROC's that may be affected.

I can think of several instances where lightning strikes some distance from a mechanical signal box has disabled the box, and at least one of those where the signaller was also disabled (thankfully, temporarily).

 

[LAX54]

My understanding (from publicly available information, and as mentioned in a previous post) is that effectively resilience has been reduced to control costs, i.e. that it was 'too difficult' (for which I would read mostly 'too expensive', but also in some case ineffective) to provide the originally planned level of resilience (operations transferred fairly swiftly to another ROC).

Note I'm not saying this was a good or bad decision; the level of resilience of individual ROCs may be such that it would be extravagant to provide the inter-ROC resilience originally envisaged. Only time will tell if this was the right call. For example, a ROC incident might have the same impact as a full-on TOC strike for a couple of weeks, but if that happens once in 20 years it might be reasonable to accept that risk.

However I do find it disappointing that it proved unfeasible to provide that level of resilience at a reasonable cost given that it was promoted (as far as I remember) as one of the main benefits of the the ROCs project in the first place.

The transfer of a ROC to another ROC, was never going to work, unless there is a big rewrite of Rules / Regs and how competence is maintained.

 

[najaB]

The transfer of a ROC to another ROC, was never going to work, unless there is a big rewrite of Rules / Regs and how competence is maintained.

In realtime, yes. But if a ROC was going to be out of commission for an extended period it wouldn't beyond the limits of reason to bus the signallers, managers, etc. to another part of the country and put them up in hotels for a week or two.

 

[edwin_m]

In realtime, yes. But if a ROC was going to be out of commission for an extended period it wouldn't beyond the limits of reason to bus the signallers, managers, etc. to another part of the country and put them up in hotels for a week or two.

Along the lines of what I posted above, I think it would be possible to re-create a ROC within a week or so from spare parts and creating comms links to a new location, and if it was a planned changeover the downtime would be less than that. Seeing as there probably won't be a suitable empty building ready for use, it would probably be better to build a temporary one near the existing facility so the staff could just travel to work as normal.

 

[najaB]

...it would probably be better to build a temporary one near the existing facility so the staff could just travel to work as normal.

I think the idea was that each ROC would be a bit bigger than it needed to be, with a few more workstations than it needed. The alternative would be to just build one extra ROC that was only used occasionally, but was swapped out most of the time.

 

[edwin_m]

I think the idea was that each ROC would be a bit bigger than it needed to be, with a few more workstations than it needed. The alternative would be to just build one extra ROC that was only used occasionally, but was swapped out most of the time.

If so it doesn't seem to have been thought through. If there was a bit of extra space in each ROC for this purpose then the functions of the ROC that was out of use would have to be split between multiple sites. This would make this more difficult both technically (systems normally adjacent would have to have long-distance communication links) and operationally (staff accustomed to working together would be split up).

The other complication is that the data that configures this equipment is generally specific to one manufacturer. There are some exceptions such as various computer-based interlockings accepting traditional SSI data, but even there I fancy some modification or at least testing would be needed to port the data onto something from another supplier. So the substitute ROC would need hardware identical to the one being replaced, which means a single ROC would need a significant setup period to take over from one of several other sites with different kit.

 

[najaB]

If so it doesn't seem to have been thought through.

I am by no means an expert on the ROC plans, I'm just repeating what I've been told/read. The issues you raise may well be why the plans were dropped.

 

[deltic08]

Quite right. Far to much info given away to people who don't need it or have a right to expect it.

All I will say is that people here seem to think that their brainwave is the first time the idea has been considered!

If a ROC burns down the impact would be the same as if the main ATC centre in the country burnt. Does that also worry you?

You obviously do not know how ATC works.

If a ROC was put out of operation for any reason everything under its control would stop for hours or days. If the ATC centre at Swanwick goes down, a limited service can continue from overlapping local airports and RAF stations.

 

[DarloRich]

You obviously do not know how ATC works.

If a ROC was put out of operation for any reason everything under its control would stop for hours or days. If the ATC centre at Swanwick goes down, a limited service can continue from overlapping local airports and RAF stations.

So if swanwick was to burn down this evening control would seamlessly transfer to another location and not interrupt services?

What about prestwick? Does that not deal with over ocean control? What if that goes up in flames this evening?

You are right I have no idea how atc works in any detail. From what I understand ( which may be wrong!) we have 2 main locations (mentioned above) that control our airspace with one dealing with control out into the Atlantic. I am unaware if there are duplicates or alternatives that could cover for.loss

Obviously we have local control for airport arrivals and departures but I have no idea how strong there monitoring and communication is. Also we clearly have military installations but I know nothing about them beyond ccf flying!

 

[najaB]

So if swanwick was to burn down...

What about prestwick?

I thought Shanwick was Shannon-Prestwick? Meaning that without Prestwick there is no Shanwick.

 

[diffident]

So if swanwick was to burn down this evening control would seamlessly transfer to another location and not interrupt services?

What about prestwick? Does that not deal with over ocean control? What if that goes up in flames this evening?

You are right I have no idea how atc works in any detail. From what I understand ( which may be wrong!) we have 2 main locations (mentioned above) that control our airspace with one dealing with control out into the Atlantic. I am unaware if there are duplicates or alternatives that could cover for.loss

Obviously we have local control for airport arrivals and departures but I have no idea how strong there monitoring and communication is. Also we clearly have military installations but I know nothing about them beyond ccf flying!

Swanwick has failed on a number of occasions in recent history. Yes you are right that in normal circumstances, it would control airways navigation through the sectors and high altitude stuff. However in the event of its failure, airports would extend their terminal areas, plus there are contingencies for high altitude and and sectors to go off to Eurocontrol in Brussels.

 

[najaB]

I thought Shanwick was Shannon-Prestwick? Meaning that without Prestwick there is no Shanwick.

Ah, my confusion is resolved. By Swanwick your mean NATS (which is located at Swanwick), rather than the Shanwick control centre.

Yes, if NATS was to go offline (again) they would transfer control to the towers and Eurocontrol (at least until March!) until it could be brought back online.

 

[boing_uk]

Discussion of national infrastructure and its weaknesses is hardly a security issue. Any reasonably intelligent belligerent is going to be able to work out the simplest ways of causing the most about of damage for the least work.

Much of the nations infrastructure is the same. A few critical locations (for instance NG switching/substations), gas compression stations, mains water infrastructure (Thirlmere Aqueduct for example) are all examples where a single act (either intentional or otherwise) will push surrounding infrastructure beyond capacity and have immediate short term (months) effects.

Let us hope that no-one gets the idea of a co-ordinated attack on multiple different infrastructures.

 

[boing_uk]

The biggest security risk to the railway will of course be from electronic attack from the internet.

I was reading an article today about subsidiary device security and how it would only take corruption of sensors or other lower-end devices to cause significant disruption to the larger “machine”.

As an example, one would hope that there is a significant level of authentication between devices on the digital railway not only to prevent changes of configuration remotely, but also sensor parameters and control messages are coming from where they’re supposed to. For instance, a virus like Stuxnet, configured to infiltrate and corrupt, say, axle counters or some other extensively used device. Significant disruption potential, potentially easy to deploy if proper security arrangements aren’t in place.

 

[AndrewE]

The biggest security risk to the railway will of course be from electronic attack from the internet....
As an example, one would hope that there is a significant level of authentication between devices on the digital railway not only to prevent changes of configuration remotely, but also sensor parameters and control messages are coming from where they’re supposed to. For instance, a virus like Stuxnet, configured to infiltrate and corrupt, say, axle counters or some other extensively used device. Significant disruption potential, potentially easy to deploy if proper security arrangements aren’t in place.

Which is exactly why the railway needs to be well-organised with in-house technical departments specifying almost all work (and components) and controlling and inspecting it down almost to the last detail.
An example that occurs to me might be level-crossing CCTV cameras, where lots of little firms are able to supply and fit them, and "dynamic new management" (or the bean-counters) might insist on a lower-price job.

 

[Bald Rick]

Discussion of national infrastructure and its weaknesses is hardly a security issue. Any reasonably intelligent belligerent is going to be able to work out the simplest ways of causing the most about of damage for the least work.

Quite so.

However there are also plenty of belligerents out there with intelligence levels that are best described as ‘below par’, and we don’t want to give them any ideas.

 

[DanDaDriver]

Quite so.

However there are also plenty of belligerents out there with intelligence levels that are best described as ‘below par’, and we don’t want to give them any ideas.

If they listen to a lot of the stuff posted on here they’ll come to the conclusion they don’t need to do anything.

 

[AndrewE]

If they listen to a lot of the stuff posted on here they’ll come to the conclusion they don’t need to do anything.

Do you mean "if they ever travel by train they will quickly conclude that they don't need to do anything more in this field to undermine the UK economy, as the railways are currently wrecking quite a lot of people's lives as it is?
The DfT seem to have a death-wish, or a hatred of rail. Micro-managed (and inadequate) franchise staffing levels - especially given the training needs on new stock (if it ever arrives), franchise contract clauses forcing conflict over DOO, do they actually want rail to play its part in moving people round the country?

 

[cjmillsnun]

Quite right. Far to much info given away to people who don't need it or have a right to expect it.

All I will say is that people here seem to think that their brainwave is the first time the idea has been considered!

If a ROC burns down the impact would be the same as if the main ATC centre in the country burnt. Does that also worry you?

I suspect both Network Rail and NATS have some form of business continuity management to allow at least a limited service should their main centre be out of action for an extended period.

I also work with critical national infrastructure and our control room is duplicated at another location. That is government mandated and is in our safety case.

 

[High Dyke]

I think the idea was that each ROC would be a bit bigger than it needed to be, with a few more workstations than it needed. The alternative would be to just build one extra ROC that was only used occasionally, but was swapped out most of the time.

That may have been the idea, but it has been suggested that York ROC isn't big enough and there is little scope for enlargement.

As you may be aware part of Kings Cross PSB has been re-controlled to York ROC. Now imagine you are stood at Kings Cross station waiting for a train to Cuffley, for example. There's an incident affecting York ROC and services have been halted. As a member of station staff can you really get a member of public to understand why their train home has been affected by a signalling incident at York - nearly 200 miles away. No? Neither could Andrew Haines, the CEO of Network Rail. He's also spoken of the 'eggs in one basket' scenario.

 

[DanDaDriver]

That may have been the idea, but it has been suggested that York ROC isn't big enough and there is little scope for enlargement.

As you may be aware part of Kings Cross PSB has been re-controlled to York ROC. Now imagine you are stood at Kings Cross station waiting for a train to Cuffley, for example. There's an incident affecting York ROC and services have been halted. As a member of station staff can you really get a member of public to understand why their train home has been affected by a signalling incident at York - nearly 200 miles away. No? Neither could Andrew Haines, the CEO of Network Rail. He's also spoken of the 'eggs in one basket' scenario.

I completely agree. But, where do we draw the line?

A small box with a distant, home and section every three miles? PSB at a major station? ROC for each major route? One big super ROC?