Work in healthcare. We have generator support for the whole site and fuel for 7 days. We have had instances in the past with the generator not starting when required. There’s a monthly test now, running for 3 hours. When the monthly test was first started we had a lot of issues which we had to log and report. Gradually these have been reduced to the point where there are very few problems now.
You need to test generator backup. If it causes problems then don’t stop testing, fix the issues.
I'd say at this point, having basically shut down the network over much of the North of England in the run up to the evening peak, that ought to be enough scale. And frankly given that it's been shown to have been such an issue, someone, somewhere should have asked the question "what if the backup fails?"
As I said earlier, someone has a very difficult day coming up tomorrow.
As said by another poster:
Manchester ROC has 2 independent 6.6kV DNO intake feeders fed from a primary substation either of which can feed the entire building by closing the relevant bus bar couplers on the HV panels ( there are some reports of a large part of Ardwick also bring without power) do sounds like it’s a SPENWL issue.
I can't speak about the Manchester ROC, but I can tell you about the arrangements for the PSB (Power Signal Box) where I was based.
Keep in mind that this PSB where I was based did not have any electrified lines.
There was a "guaranteed" feed from the main sub-station for the city (the DNO supply). Here "guaranteed" just means that there were no other customers fed from the same cable. This was an older installation. In newer installation, there are often two independent feeder cables.
An online UPS was provided to maintain power for several minutes. This should be enough time to allow the standby diesel generator to start. The control system for diesel generator was set up to automatically start it if power was lost for more than about 20 to 30 seconds. The diesel generator was designed to be ready to take the load in around 1 minute.
I can't remember the frequency (I think it was three monthly), but "crash" testing of the standby diesel generator (where the incoming mains is switched off and you check that the UPS can supply the load, that the generator automatically starts and that it takes the load) was part of the maintenance schedule. Although, because permission was needed from the signallers, this did not always happen. In which case the generator had to be run off-load. The test run was typically around one hour IIRC.
The fuel tank that feeds the generator was suppose to hold enough fuel for at least 24 hours. The fuel tank may of course be bigger than this. If the tank only holds 24 hours worth, there should also be a second fuel tank with more fuel.
In addition to the automatic starting system, standby diesel generators also have a manual push button start system and some may also have an emergency starting system.
Everyone seems to be "blaming the railway".
Has anyone considered that this surprising combination of issues could actually be due to external factors e.g. hackers / foreign interference?
I seem to remember one incident last year was quietly tentatively blamed on Russian interference - could this be a scaled up attack with the aim of causing disruption?
The railway uses closed control systems for the signalling system and the power system for the signalling system. In order to hack it, you have to be on railway property.
Oh we do...
But this has been happening since the 60s with PSBs etc.
This is true, however, some 1960s/1970s PSBs did have emergency through routes for the remote interlockings, some of which would still work during a total loss of mains power to the PSB. How? Because the through route control circuit was fed from a 50V battery at the PSB. Further, some remote interlockings had emergency panels that could be switched in.
None of this helped for the area of the railway that had lost power, but it could have kept some of the area controlled by the PSB operational.
But of course, this cost money, so, because it was hardly ever used, some of the emergency panels were taken out of use.
No secondary site for Saltley that I know of, not sure where you would put it.
Alas, with computerised signalling, because it's not a design requirement, it's much harder to have a secondary site, and because the interlockings are now normally at the ROC/signalling centre, there are no remote interlockings to control...
Is Network Rail a basket case?
By which I mean guilty of putting more and more eggs in the basket until it breaks.
Case 1) the current disgrace - as commented by others earlier, full testing with switchover onload and prolonged running in that state is mandatory.
If you decide to replace staff in signal boxes with levers, with digital stuff, you need to be more resilient than they were over the whole area in question.
My background in broadcasting involved very regular tests involving killing the mains and testing that the diesels and battery UPS covered the outage and that nobody noticed it on the output!
Case 2) bigger basket, more eggs, more people affected and more frequent - OHL!
There seems to be precious little understanding of 'Single Point of Failure' (SPF), whilst there is much excitement and whooping about new areas going under the wires, OHL has an enormous disaster disruption potential that steam or diesel locos just don't have.
Reducing the number of Thunderbirds as opposed to increasing them seems to indicate the basic lack of understanding of risk that is being undertaken.
There is an issue of professionalism here, if you're asked to design and implement something that you don't feel has the necessary reliance considering the potential SPF consequences, don't do it, don't complain about budget constraints - yes of course that's difficult.
Yes, the bigger you build it, the harder it falls.
See my comments earlier in this post about the power supply arrangements where I was based.
As I understand it, the track sections would show as if there's a train is there if there's no power, but when the power comes back, they will go back to clear if there's no train on it.
Is that right?
Nothing will show if the Control System has no power! The screens will be totally blank, no track layout, zilch.
I think you are thinking of a local trackside power failure, when what you say would be correct. But this was a failure of the control centre. Once it comes back up, it should show the state of the track circuits as currently reported by the interlockings. And that will depend on the type of interlocking and indication system, and the state they are in. And even once the track circuits are being indicated, the system may have lost all the train descriptions, in which case the TD displays will be blank, and so the signallers will have no clue what, if any, train or trains is in each section showing as occupied.
Further to what Belperpete says...
Track circuits will return to working normally when power returns (if power has even been lost, some remote areas will be powered by different supplies, not everything is powered from the same supply as the ROC/signalling centre). But if axle counters are being used, if their battery backup could not sustain their "state" (section clear or if occupied, the number of axles counted in), they will forget their state and will show the section as occupied until such time that they are manually reset.
You are assuming that the system would automatically switch back to the DNO supply as soon as it comes back on. On systems that I am familiar with, you don't switch back until you are sure that the supply is stable and can be relied on. If so, the system would switch to the generator when the DNO supply failed, and would remain switched to the generator even though the DNO supply came back on, until a technician manually triggered switching back. So how long the DNO supply was off for would be irrelevant.
The issue would be how long it took to get the generator up and running, and how long the UPS maintained the supply for. Did it take the generator too long to get on line, and/or did the UPS not maintain the supply for as long as it should have?
The control systems for railway standby diesel generators will indeed not automatically switch back to the mains (DNO) supply. They have to be manually switched back. And anyway, transferring back requires synchronising the phase of the generator to the exact same frequency and phase of the mains supply. You can't just parallel two different AC supplies. If the frequency and phase are different, there will be a big bang and the generator will stop dead and all power to the load will be lost...
Having said that, if the standby diesel generator develops a fault, some control systems will automatically attempt a switch over back to the mains (DNO) supply. But this is not seamless as the load loses power before the switchover occurs.