Tunnel work on 10th September cut a cable

[endecotp]

Dear All,

I don't think I saw any mention of this incident, does anyone have any idea where it could have been? Which tunnel was being worked on on 10th September?

Airline 'in talks' with Kyndryl over massive IT outage

Delays and cancellations thought to have cost Aer Lingus millions

www.theregister.com

The story is that Aer Lingus suffered disruption due to a cut fibre optic cable and a co-incidental failure of the backup system. The cable cut was apparently due to railway tunnel works:

Their ISP in the UK had a major failure on that day, early Saturday morning, where there was unrelated construction work which damaged a fiber optic cable – and that cable would provide connection to our systems. The cable wasn't repaired until 5.30 in the afternoon. It was a damage that occurred just after eight o'clock in the morning."

The Aer Lingus exec pointed out that their data had been mirrored to two separate sites, a datacenter in Manchester, and the second one in Birmingham by their IT services provider, and that the lines had been replicated into both. "So it should have been more resilient than it proved to be on the day."

It is not known where the fiber optic cable was located, but the CEO told the committee the cut to the line had occurred during rail tunnel work before a separate unconnected hardware error happened on the backup line.

The fact that the site (of the damage) was a railway line meant it took them some time to get on site after the construction machinery broke through the casing

 

[Gloster]

I presume that this was in Ireland, where there are (I think) only fifteen tunnels, most of which are on the Rosslare line south of Bray. It is mostly likely Phoenix Park Tunnel or the one on the North Wall line; the one at Dalkey is at outside possibility, or maybe even Cork.

 

[endecotp]

I presume that this was in Ireland

No: "Their ISP in the UK had a major failure on that day"

The locations mentioned are Manchester and Birmingham but network cables often take surprising routes.

 

[Gloster]

No: "Their ISP in the UK had a major failure on that day"

The locations mentioned are Manchester and Birmingham but network cables often take surprising routes.

Ah, I missed that bit. Still, an interesting bit of nostalgic checking.

 

[noddingdonkey]

If you've got two data centres, 80 miles apart, and cutting one fibre optic cable takes them both offline, you've probably screwed up your resilliance planning somewhere.

 

[Elecman]

It was 2 separate faults not interrelated

 

[D6130]

I wonder whether this incident involved the former Mercury Telecoms cables which, IIRC, were laid alongside the West Coast Main Line sometime back in the 1980s and have now been put to other uses?

 

[Elecman]

Wonder if that was Beechwood tunnel works near Berkswell?

 

[Mcr Warrior]

The online article quoted by the OP states, in particular...

It is not known where the fiber optic cable was located, but the CEO told the committee the cut to the line had occurred during rail tunnel work before a separate unconnected hardware error happened on the backup line.

...which doesn't really help narrow down the location of the incident.

 

[gingertom]

It was 2 separate faults not interrelated

unfortunately these things do happen. Some 20+ years ago I had a similar incident at my workplace. There was a fire in an ancient building in the "old town" of Edinburgh; some of you here might recall it. This incident damaged some telecomms infrastructure and I was advised that we had "lost resilience" due to the incident. The backup plans were working fine, but sometime later before this incident was resolved there was a water main rupture in Glasgow: Paisley Road West. During the excavation in order to carry out repairs to the water main the contractors managed to damage telecomms cables. Guess what: this carried our backup systems. Totally out until the water main was fixed and the telecomms guys made their fibre optic cables good.

 

[swt_passenger]

Would the people running internet infrastructure want the public to know exactly where their stuff is most vulnerable?
I suspect “somewhere between Manchester and Birmingham might be as detailed as you’d ever get…

 

[Taunton]

Just to put this in context, it is by no means unknown for a provided alternative computer facility to fail at the odd point it is needed, due to the changeover not having been set up properly in the first place, and/or not having been adequately and regularly tested. So it can be just laying in wait for the first time it is required. Has happened time and again. Sounds like what happened here.

A classic example was a backup diesel generator as automatic backup to the main power supply, all installed and periodically tested changing over and back. When actually required it stopped after a short while because it ran out of fuel, nobody in the procedures having ensured that it was maintained fuelled, and the testings had used much of what it was delivered with.

 

[Glasgowbusguy]

unfortunately these things do happen. Some 20+ years ago I had a similar incident at my workplace. There was a fire in an ancient building in the "old town" of Edinburgh; some of you here might recall it. This incident damaged some telecomms infrastructure and I was advised that we had "lost resilience" due to the incident. The backup plans were working fine, but sometime later before this incident was resolved there was a water main rupture in Glasgow: Paisley Road West. During the excavation in order to carry out repairs to the water main the contractors managed to damage telecomms cables. Guess what: this carried our backup systems. Totally out until the water main was fixed and the telecomms guys made their fibre optic cables good.

Part of the issue is that no one seems to know what is exactly where , we had major road works outside my work over the summer and the guys traced all the drains and cables before they started digging and what they found was totally different from the map the council had.
They also found that drains that where ment to be coming in through the front of our building actually spurred off and came around the side and in via the rear of the building.

 

[Ediswan]

A classic example was a backup diesel generator as automatic backup to the main power supply, all installed and periodically tested changing over and back. When actually required it stopped after a short while because it ran out of fuel, nobody in the procedures having ensured that it was maintained fuelled, and the testings had used much of what it was delivered with.

To add to the complications, diesel can only be stored for a limited length of time before you should replace it. Variable, but some suggest as short as 18 months.

 

[gingertom]

To add to the complications, diesel can only be stored for a limited length of time before you should replace it. Variable, but some suggest as short as 18 months.

Some of my employer's sites had diesel generator backup. There was a strict maintenance regime in place to ensure that the diesel would start and run and provide power in the event of an outage. After some 20 years it was decided that, at one particular site, the diesel had accumulated sufficient engine hours to be in need of a more serious service, over and above oil and filter change, swapping the fuel etc, and required the cylinder heads to come off for a decoke and a valve regrind. Whilst the heads were off the mains power died....

 

[Ediswan]

Some of my employer's sites had diesel generator backup. There was a strict maintenance regime in place to ensure that the diesel would start and run and provide power in the event of an outage. After some 20 years it was decided that, at one particular site, the diesel had accumulated sufficient engine hours to be in need of a more serious service, over and above oil and filter change, swapping the fuel etc, and required the cylinder heads to come off for a decoke and a valve regrind. Whilst the heads were off the mains power died....

The data centre I once worked in had that covered, in principle. There were two diesel generators, each of which could in principle handle the full load (mounting blocks for a third). There were also connectors and hard standing for external standby generators. They also had a (somewhat leaky) diesel header tank high up on an internal wall to improve the chances of the main diesels starting.

The only **** up was a planned test. Disconnect the mains supply and see if the batteries, and then diesel generators, cut in. They did, perfect. Now switch back to mains. Oops. The system logic had been tweaked since originally installed and would now only re-connect the mains if the mains was already connected. Chicken, meet egg. Some live electrical work required (not me). There was no interruption to sevice, which is the underlying goal.

 

[tellytype]

Some of my employer's sites had diesel generator backup. There was a strict maintenance regime in place to ensure that the diesel would start and run and provide power in the event of an outage. After some 20 years it was decided that, at one particular site, the diesel had accumulated sufficient engine hours to be in need of a more serious service, over and above oil and filter change, swapping the fuel etc, and required the cylinder heads to come off for a decoke and a valve regrind. Whilst the heads were off the mains power died....

Sounds like a situaiton I narrowly avoided at Worldpay when I took over running their datacentres.

Large diesel in the car park for the backoffice functions (and core routing for the whole company!) - diesel polished as had been sat forever...once that was done I purged the crap out of the lines (and gave it a change of filters....client loved me because I did it myself & ran great, changeover unit in main data floor, awesome...cables from the diesel....juuuust floating around at the back of the C/OU not connected. (Facepalm!)

Its all closer to falling apart than you think!!

 

[TheSeeker]

In university we studied using laser reflection readings to discover the exact point of a fiber cut. So technically possible to find the geographic point of the incident with some accuracy.

 

[tellytype]

In university we studied using laser reflection readings to discover the exact point of a fiber cut. So technically possible to find the geographic point of the incident with some accuracy.

Yep. OTDR. Optical Time-Domain Reflectometry. I own a machine that does just what you describe. Its a network operator function to do this though & carrier equipment may well have the ability to do an OTDR run if it loses light on the link, although its likely this cable was carrying serveral DWDMs down each fibre & we dont know what size of fibre it was/is (144 is possible!!) .

As long as the pulse width was short enough (if test point was close in) or you were far enough away its easy to pinpoint to within inches where the fault is AND from the trace see what type of fault it is. See this example

 

[ejstubbs]

There was a fire in an ancient building in the "old town" of Edinburgh; some of you here might recall it.

Are you referring to the Cowgate fire that took out the Gilded Balloon? This year will be the 20th anniversary:

On 7th December 2002 a fire in the Cowgate destroyed part of the heart of Edinburgh’s old town. Although the fire brigade arrived within minutes the fire had quickly spread to the South Bridge before consuming the Gilded Balloon. The fire burned for four days and a total of eleven buildings were destroyed in one of the worst fires Edinburgh’s living memory.

Another fire-related incident in Edinburgh a few years later: the bank I worked for had paid BT for diverse routing for its leased lines to London for the CHAPS service. One fine day both lines went down at the same time. In the subsequent "meeting without biscuits", BT revealed that both lines were routed through the exchange at Slateford, which had suffered a major fire. When pressed on the matter, with reference to the fact that the bank was paying them good money for diverse routing, their response was: "Well, the equipment was on different floors..."

== Doublepost prevention - post automatically merged: 13 Oct 2022 ==

A classic example was a backup diesel generator as automatic backup to the main power supply, all installed and periodically tested changing over and back. When actually required it stopped after a short while because it ran out of fuel, nobody in the procedures having ensured that it was maintained fuelled, and the testings had used much of what it was delivered with.

Another of my past employers had the diesel generators on the roof. All fuel had to be carried up the stairs in jerry cans; transporting it in the lift was not permitted. One fine day (again) the Scottish Power substation in the car park blew up (literally, though thankfully not too violently). The premises guys spent several days shuttling diesel up to the gennies before SP got a whole new set of substation equipment installed and working (yes, it was that bad). There was one saving grace: the building was right next door to a Sainsbury's petrol station - and their pumps were still working. Presumably someone bagged a whole lot of Nectar points out of that...

 

[deep south]

I worked with a company that had the standby diesel generators next to the office building but staff complained about the noise and smell during the regular testing, so management decided to move them to the other side of the car park away from the building. A trench was dug for the pipes and cables and an electric pump installed for supply from the tanks which were left in the original position to save money.
Of course, no one realised that the pump was driven from the mains until several minutes into a power cut when the generators went very quiet!

 

[Taunton]

I'm afraid it's the old story of the astronaut to the moon being asked what it feels like when ready to go, and replying "I'm sat on top of 5 million components, all bought from the bottom bidder". The person with the cheapest quote is often the one that has left the most functionality out. Things that are not in regular use, like backups, are the ones that suffer from cheapening to stay inside budget. Plus they don't get tested or checked as well as the main system. I've found previously well documented approaches for the cutover to the standby, but nothing for how to change back from standby to main system when it's been fixed.

 

[gingertom]

Are you referring to the Cowgate fire that took out the Gilded Balloon? This year will be the 20th anniversary:

That's the one.

 

[najaB]

I think I've mentioned in another thread this anecdote from a previous employer.

They had a diesel backup generator which was regularly tested, and worked perfectly during a power cut. Unfortunately, the transfer switch did not and fused in the 'generator' position. Which meant that when the (six?) hours of fuel ran out the call centre went dark. Took two days to get everything back up and running.

 

[MadMac]

There is a critical junction on the LA Metro fitted with a backup generator for the signals and points etc. Never, to my knowledge, been maintained or tested since installed. Signals say it’s a Traction Power responsibility: Traction Power say it’s a Signals responsibility. The “pi**ing contest“ has been going on since 2011…..

 

[gingertom]

There is a critical junction on the LA Metro fitted with a backup generator for the signals and points etc. Never, to my knowledge, been maintained or tested since installed. Signals say it’s a Traction Power responsibility: Traction Power say it’s a Signals responsibility. The “pi**ing contest“ has been going on since 2011…..

all that's needed to sort the politics of ownership out is for there to be an outage and the diesel doesn't kick in.

 

[MadMac]

all that's needed to sort the politics of ownership out is for there to be an outage and the diesel doesn't kick in.

Bingo!