Nats fallen over again

[Tetragon213]

Most won't believe the many, many attacks on infrastructure were Russia unless there is a verified signed document by Putin saying "I did it - Yours, Putin".

edit: Crazy how many people refuse to accept we're in a hybrid war with Russia and have been for a long time. Yes I know there are other causes for things

Oh, I can do you worse.

Had a housemate who outright defended Russia attacking us, with some choice words that boiled down to justifying Russia attacking other nations.

Some people (we would call them Tankies) are genuinely beyond hope.

 

[Edsmith]

There is still a Right to Care in place. If there's enough stock to give out.

That's it, if there is enough stock to give out. The airlines have had the rug pulled from under them and must be in a state of disarray.

 

[nlogax]

Interesting to read that Martin Rolfe still commands the confidence of No.10 to at least report back on the outcome of the investigation into this week's mess. My guess is once he's shared the root cause and the investigation's findings he'll cease to be CEO of NATS.

https://www.standard.co.uk/news/uk/...f-executive-air-traffic-control-b1296285.html

“[Heidi Alexander] asked him to carry out a full investigation into what happened to make sure that lessons are learned, so that this doesn’t happen again.
“She set a deadline for them to report back in a week’s time.”
Asked whether Ms Alexander and Andy Burnham still have confidence in Mr Rolfe, the Prime Minister’s spokesman added: “It’s self-evident that she wants answers and the full investigation to be reported back in a week’s time, and she’s got confidence in him to deliver that.”

This statement appeared to fall short of full confidence in Mr Rolfe.
Pressed whether this meant the Government’s confidence in the Nats’ chief was conditional on the outcome of the investigation, the spokesman replied: “We have confidence in him and he’s been tasked with the investigation, which will report in a week’s time.”

 

[Bantamzen]

Interesting to read that Martin Rolfe still commands the confidence of No.10 to at least report back on the outcome of the investigation into this week's mess. My guess is once he's shared the root cause and the investigation's findings he'll cease to be CEO of NATS.

https://www.standard.co.uk/news/uk/...f-executive-air-traffic-control-b1296285.html

After 3 major fails in 3 years, he should be packed off to the job centre. If most ordinary folk where at the heart of such a monumental cock up* we'd be out on our ear. But this is UK plc, where the consumer doesn't matter so long as the profits roll...

(*I know that the suspicion is a cyber attack from some quarters, but even if it was this there should have been better protection / mitigation in place for such a critical system)

I got caught up in this yesterday trying to fly out to Crete from Manchester. The really galling thing was when we finally found out we were being cancelled and the holiday binned, I could literally see the craft allocated to us on one of the remote stands. Worse still was the complete breakdown in comms, I know that staff on the ground knew little more than us, but when the time came for cancelling multiple Jet2 flights hundreds of use were just walked out to Border Control without anything more than "it's been cancelled" from staff. Were we to wait landside for a later flight, or one in the morning? Were our holidays canned? Nothing was clear until emails started to arrive confirming that we had our holidays cancelled and will eventually get a refund. This was around 9pm last night, in the baggage hall with thousands of increasingly angry passengers trying to find out if they'd even get their bags, let alone which carousel scrum you had to go to. Thankfully it "only" took 2 hours from leaving departures for me to get my suitcase, the hall was still rammed when I shoulder barged myself to the exit.

I've already ranted about this on social media, so I'll keep this short. It is time to bring key infrastructure systems like NATS back into full public control, and more importantly accountability. One failure is a failure too many, 3 is criminal. And more importantly, situations like this should trigger compensation not just refunds for cancellations. It's an absolute disgrace that some people have to wait for days to get flights home as some poor souls are having to do, or just simply having to mill about increasingly packed and stressful areas like the baggage hall whilst desperately trying to work out which belt their luggage may appear. Perhaps if such measures were in place, some minds might focus a bit more.

 

[renegademaster]

The problem with going on too much about foreign adversaries is ultimately there's not actually a huge amount you can actually do about them other than not have vulnerabilities for them to exploit in the first place, which is prevented by good quality control and security practices that are a good idea regardless.

 

[Stossgebet]

Even if it was malicious action, Russia being a top 3 suspect if so. People seem to forget that our government wouldn't be shouting about it. For evidence, see how a (potentially made up) Russian intermediary was blamed for ordering the firebombing of then Prime Ministers Keir Starmers property. And the during the court case, the judge specifically ruled out questioning the underlying motivations behind the firebombing/arson attacks. And the whole shameful scandal was barely news.

 

[Sun Chariot]

DDOS cyber attacks are just as likely to come from nations other than Russia.
I was at the organisation when a DDOS attack occurred. The identification and preventative action happened within a very short space of time: seconds. Contingency operations kicked in within minutes.

August 2024's outage was due to, iirc, bad data within a daily routine interface from France.
So I am more inclined to think this will be a root cause where a correct failover was not identified or acted upon in a timely manner; thus compounding and lengthening its effect.

Whatever the findings, I hope the operation emerges more resilient. I also hope that the Public-Private* structure aids, not hinders, the decisions or investments recommended.

* UK Government owns 49% of NATS; and it holds a "golden share" with special voting rights to block decisions—such as protect national security interests.
The Airline Group (a consortium of airlines) owns 42% NATS.
London Heathrow Airport Holdings owns 4%.

 

[Bald Rick]

I got caught up in this yesterday trying to fly out to Crete from Manchester. The really galling thing was when we finally found out we were being cancelled and the holiday binned, I could literally see the craft allocated to us on one of the remote stands. Worse still was the complete breakdown in comms, I know that staff on the ground knew little more than us, but when the time came for cancelling multiple Jet2 flights hundreds of use were just walked out to Border Control without anything more than "it's been cancelled" from staff. Were we to wait landside for a later flight, or one in the morning? Were our holidays canned? Nothing was clear until emails started to arrive confirming that we had our holidays cancelled and will eventually get a refund. This was around 9pm last night, in the baggage hall with thousands of increasingly angry passengers trying to find out if they'd even get their bags, let alone which carousel scrum you had to go to. Thankfully it "only" took 2 hours from leaving departures for me to get my suitcase, the hall was still rammed when I shoulder barged myself to the exit.

Sounds like your issue is more with MAG and Jet 2. Perhaps it’s their CEOs who should be hung out to dry.

 

[Sun Chariot]

Sounds like your issue is more with MAG and Jet 2. Perhaps it’s their CEOs who should be hung out to dry.

+1 to this.

NATS does air traffic management in Upper Airspace and in Lower Airspace and for some UK airports' Terminal Control.

 

[Tetragon213]

The problem with going on too much about foreign adversaries is ultimately there's not actually a huge amount you can actually do about them other than not have vulnerabilities for them to exploit in the first place, which is prevented by good quality control and security practices that are a good idea regardless.

If someone is determined enough, and has sufficient resources, they will find a way through even the most elaborate of defences. It has been noted that the timing is rather odd, as it comes just hours after sanctions were placed on West Bank settlements; this has led to some humorous (and some less light-hearted) discourse about "Mossad hackers"; see the FlightRadar24 Twitter post...

 

[BananaRepublic]

If someone is determined enough, and has sufficient resources, they will find a way through even the most elaborate of defences…….

They can’t find a way through where it’s not connected to any publicly, or wider www networks.
In other words, they cannot reach which is unreachable.

The cause of this outage is said to be known and has been fixed.
The NATS systems are a combination of very complex individual systems working together.
The FDP (flight data) element is crucial, as it “feeds”, underpins and interacts with the other systems.
The levels of redundancy are far higher and more resilient than almost anything on the planet, but as with any massively complex environments, there will be latent weaknesses, particularly when dealing with FDP.

The previous significant outage occurred due to a subsidiary system (basically an interpreter) falling over in a million to one scenario (bad flight data).
We will be told where this issue occurred in due course.

No change of management or oversight can alter the fact that the NATS system is a very complex and highly advanced beast.
The radar tools and controller workstation tech is class leading, however the underlying FDP system is ancient, but very capable and usually highly dependable.
They’ve been working on a replacement FDP, along with European colleagues, for more than a decade, but it is has proved to be a very difficult process.

Failures like this may be catastrophic for airline operations and for passengers, but they occur safely and the operation is slowed down, or closed down in a controlled manner.
Unfortunately the knock on effects go on for a lot longer than the hour or two that contingency operations are in force.

The skies are so busy, that when the highly complex ATC systems have failures or outages, it’s impossible to maintain the normal level of air traffic movements. Fact.

 

[renegademaster]

If someone is determined enough, and has sufficient resources, they will find a way through even the most elaborate of defences

If this was true, they'd also hack something more consequential like Visa Or MasterCard

 

[Watershed]

The previous significant outage occurred due to a subsidiary system (basically an interpreter) falling over in a million to one scenario (bad flight data).

The flight plan was valid, the system just couldn't cope with it. Given the number of flights that take place each year, "one in 15 million" (which is the figure that NATS gave as an excuse) really means "every 10 years". I don't think that's a remotely acceptable level of reliability for such a crucial piece of national infrastructure

The levels of redundancy are far higher and more resilient than almost anything on the planet, but as with any massively complex environments, there will be latent weaknesses, particularly when dealing with FDP.

It's clearly not very redundant or resilient at all if it can go down like this for such a long time.

We already saw with the flight planning interpreter failure that if a particular flight plan triggers a bug in the code, the backup system could lock up/fail in just the same way because the systems aren't designed to skip the problematic flight plan for manual intervention, and continue working otherwise.

I wouldn't be surprised if it's a case of a similar type of failure in terms of poor systems design.

 

[mpthomson]

The obvious possibility here is that a certain vodka drinking nation might have been busy...

Ryanair has said that its a repeat of the issue in 2023 (and that was nothing whatever to do with Russia, or Israel, which is the latest theory from the crazies, in response to Milliband's announcement).

 

[pitdiver]

After 3 major fails in 3 years, he should be packed off to the job centre. If most ordinary folk where at the heart of such a monumental cock up* we'd be out on our ear. But this is UK plc, where the consumer doesn't matter so long as the profits roll...

(*I know that the suspicion is a cyber attack from some quarters, but even if it was this there should have been better protection / mitigation in place for such a critical system)

I got caught up in this yesterday trying to fly out to Crete from Manchester. The really galling thing was when we finally found out we were being cancelled and the holiday binned, I could literally see the craft allocated to us on one of the remote stands. Worse still was the complete breakdown in comms, I know that staff on the ground knew little more than us, but when the time came for cancelling multiple Jet2 flights hundreds of use were just walked out to Border Control without anything more than "it's been cancelled" from staff. Were we to wait landside for a later flight, or one in the morning? Were our holidays canned? Nothing was clear until emails started to arrive confirming that we had our holidays cancelled and will eventually get a refund. This was around 9pm last night, in the baggage hall with thousands of increasingly angry passengers trying to find out if they'd even get their bags, let alone which carousel scrum you had to go to. Thankfully it "only" took 2 hours from leaving departures for me to get my suitcase, the hall was still rammed when I shoulder barged myself to the exit.

I've already ranted about this on social media, so I'll keep this short. It is time to bring key infrastructure systems like NATS back into full public control, and more importantly accountability. One failure is a failure too many, 3 is criminal. And more importantly, situations like this should trigger compensation not just refunds for cancellations. It's an absolute disgrace that some people have to wait for days to get flights home as some poor souls are having to do, or just simply having to mill about increasingly packed and stressful areas like the baggage hall whilst desperately trying to work out which belt their luggage may appear. Perhaps if such measures were in place, some minds might focus a bit more.

My wife and I were in exactly the same position. We were due to fly out from East Midlands Airport at 15.05 to Fuerventura for a special holiday. We were sat on the aircraft for two hours but eventually told to return to the terminal. Where we sat for another couple of hours. Then someone noticed that our flight had disappeared from the screens. Eventually and I mean eventually we were advised the flight had been cancelled and all associated holidays as well. All we were told was we would get a refund. Then the emails and texts followed. Forgot to mention it was with Jet2.

 

[Terry Tibbs]

‘I've already ranted about this on social media, so I'll keep this short. It is time to bring key infrastructure systems like NATS back into full public control, and more importantly accountability.’
I struggle to understand the logic to first part of the quote. I do not see how or why NATS being in public control would have possibly prevented its systems falling over. I agree with full accountability and leadership taking the responsibility for these messes though. However bringing NATS entirely under public control I fear would make it as effective, efficient, dynamic and flexible as other state controlled monoliths. The levels of accountability in government run organisations is also as close to zero as possible.
UK ATC is extremely good and the level of control is world class. UK airspace (London in particular) is some of the busiest and complex in the world. I am certainly not saying it’s perfect and incidents like what has just happened should be expected - they shouldn’t. But I for one would not want NATS to be owned by the state. Certainly government oversight and regulation but not public ownership.

 

[Sun Chariot]

I agree @TerryTibbs and, in post 37, @Bantamzen , I laid out the primary owners - and influencers - of NATS:
49% UK Government, which also has power to veto decisions of other owning entites.
42% The Airline Group, a consortium of the airlines operating into and out of the UK.
4% London Heathrow Airport Holdings.

Those latter two entities - for airlines and airports - act in part as a "conscience". I think that conscience risks being lost - or at least diminished - in a fully state-owned org.

 

[Bantamzen]

Sounds like your issue is more with MAG and Jet 2. Perhaps it’s their CEOs who should be hung out to dry.

+1 to this.

NATS does air traffic management in Upper Airspace and in Lower Airspace and for some UK airports' Terminal Control.

Not really. Jet2 had the craft on the ground, but MAG couldn't provide buses to the remote stand because flights were being diverted to Manchester because of the failure. There was then a good two hour period when nothing took off from Manchester, which we told was as a direct result of the NATS failure. Sorry, but they don't get off here, regardless of the area they cover they caused chaos around the UK and Europe.

 

[Butts]

I've already ranted about this on social media, so I'll keep this short. It is time to bring key infrastructure systems like NATS back into full public control, and more importantly accountability. One failure is a failure too many, 3 is criminal. And more importantly, situations like this should trigger compensation not just refunds for cancellations. It's an absolute disgrace that some people have to wait for days to get flights home as some poor souls are having to do, or just simply having to mill about increasingly packed and stressful areas like the baggage hall whilst desperately trying to work out which belt their luggage may appear. Perhaps if such measures were in place, some minds might focus a bit more.

Unlikely as The Government would be in the chair for half the amount paid out - haven't got a great track record from previous far more serious events.

Would you bring back British Airports Authority and Nationalise BA as well ?

 

[Bantamzen]

Unlikely as The Government would be in the chair for half the amount paid out - haven't got a great track record from previous far more serious events.

Yes they would, so I would hope that would focus minds a little more. And it certainly would lead to more serious questions being asked of the people in charge, because the CEO seems almost bulletproof despite several failures.

Would you bring back British Airports Authority and Nationalise BA as well ?

Not necessarily, I'm more interested in key infrastructure being within full public control. I have ever decreasing faith in the private sector, especially with key systems.

 

[AndrewE]

Ryanair has said that its a repeat of the issue in 2023 (and that was nothing whatever to do with Russia, or Israel, which is the latest theory from the crazies, in response to Milliband's announcement).

Looks like they might be correct: "spurious data!" (https://www.theguardian.com/world/2...edly-blamed-for-uk-air-traffic-control-outage)

https://www.google.com/preferences/source?q=theguardian.com
According to the Financial Times, four people briefed on the incident claimed that the shutdown was caused by a flight plan submitted by a UK military aircraft, which caused a meltdown in air traffic control systems used by Nats which remained unfixed for several hours.
The paper also reported that Nats operations officer Kathryn Leahy called the operations managers of numerous airports and airlines, and “tensions boiled over” after Leahy informed them their flight processing system did not have a dedicated backup due to Nats being in the midst of a “complete overhaul of its systems” and because the amount of data it keeps meant duplicating it would cost “vast amounts”.

 

[Bantamzen]

Looks like they might be correct: "spurious data!" (https://www.theguardian.com/world/2...edly-blamed-for-uk-air-traffic-control-outage)

What also costs vast amounts is canning tens if not hundreds of thousands of customer's flights / holidays, pay for additional accommodation & food for stranded passengers, coaches for passengers diverted & logistics costs for those passengers whose luggage stayed on the planes whilst they were coached home (which happened at Manchester).

Frankly if this is true, and I have no reason to doubt it, then it's a shocking failure from NATS. Having a system overhaul is not a good reason for not backing up data, in fact it's more of a reason to do so. Heads really do need to roll there, starting at the top.

 

[nlogax]

Looks like they might be correct: "spurious data!" (https://www.theguardian.com/world/2...edly-blamed-for-uk-air-traffic-control-outage)

2023's problem stemmed from a flight plan originally submitted to Eurocontrol for a flight from LAX to Paris which failed validation in the Nats-owned FPRSA-R system, causing an exception which brought the system to its knees.

I'm curious about this one - would a UK military flight plan submitted via MAPP (MoD's newish flight plan submission component) require processing by FPRSA-R or are those FPs 'native' to Nats' systems?

Either way, if the military traffic FP suggestion is true then Ryanair could be forgiven for thinking it's the same problem even if a different component was involved. Rolfe needs to go.

 

[Sun Chariot]

Looks like they might be correct: "spurious data!" (https://www.theguardian.com/world/2...edly-blamed-for-uk-air-traffic-control-outage)

Just sayin' (link to my post 10 on Tuesday)

Nats fallen over again

Seems like part of the UK ATC system has fallen over again, with some airports having no departures for a while but with operations now resumed and heavily limited. Flightradar shows planes at Heathrow still at the gate 4 hours after they should have departed, plus many planes on the ground...

www.railforums.co.uk

 

[Watershed]

What also costs vast amounts is canning tens if not hundreds of thousands of customer's flights / holidays, pay for additional accommodation & food for stranded passengers, coaches for passengers diverted & logistics costs for those passengers whose luggage stayed on the planes whilst they were coached home (which happened at Manchester).

Frankly if this is true, and I have no reason to doubt it, then it's a shocking failure from NATS. Having a system overhaul is not a good reason for not backing up data, in fact it's more of a reason to do so. Heads really do need to roll there, starting at the top.

Ah but you see - those costs are borne by airlines (and uninformed/misinformed passengers), not NATS.

NATS have no apparent incentive to provide a reliable service because they carry no liability when they mess up.

 

[WelshBluebird]

2023's problem stemmed from a flight plan originally submitted to Eurocontrol for a flight from LAX to Paris which failed validation in the Nats-owned FPRSA-R system, causing an exception which brought the system to its knees.

I'm curious about this one - would a UK military flight plan submitted via MAPP (MoD's newish flight plan submission component) require processing by FPRSA-R or are those FPs 'native' to Nats' systems?

Either way, if the military traffic FP suggestion is true then Ryanair could be forgiven for thinking it's the same problem even if a different component was involved. Rolfe needs to go.

Specifically 2023s problem was due go a duplicate way point name (two different geographically distant way points that happened to have the same code). And unusual but not unheard of scenario. Such a thing really shouldn't be bringing down entire systems, though as a software engineer I am painfully aware that the devil is often in the detail of the requirements presented before the system is bought / built.

As you say definitely would be interested to know the detail of this one. Anything similar to 2023 would raise huge question marks asking if they learned anything from 3 years ago.

 

[nlogax]

As you say definitely would be interested to know the detail of this one. Anything similar to 2023 would raise huge question marks asking if they learned anything from 3 years ago.

A single flight plan causing this sort of pain has to be unacceptable regardless of the cause. Allegedly running without a secondary side makes it way worse. Although I suppose even in 2023 it didn’t matter, the failover didn’t matter as the secondary system choked on the specific FP in exactly the same way.

 

[DeverseSam]

Can the railway help out at least for the internal flights. Extra late train by Avanti or LNER? Sweep up those hoping to travel later today within the UK?

Yes it wasn’t well advertised but train tickets could be used on the rebooked flight date if proof shown.

In addition you could use your outbound train ticket as a return (home) train ticket if your outbound flight was cancelled.

 

[Starmill]

What disappoints me is that so, so many thousands people will have lost out with extra costs of not having their checked bags available to them / extra taxi and hotel bills / food and drinks which they had absolutely no choice but to pay for at airport prices after hours and hours unexpectedly without. Only a small proportion of those will go through the hassle of keeping the receipts, submitting claims and chasing if (when, for certain airlines) their airline doesn't respond, doesn't understand the claim, or incorrectly declines the claim.

The only thing that might help is if people use free resources like the money saving expert pages. The majority of the general public aren't well versed in what the right to care covers, when it applies and how to claim for it. If they do have an understanding, they may still quite reasonably misunderstand the difference between compensation for delays and costs incurred under right to care.

 

[Terry Tibbs]

The service NATS provides is reliable. There are ATC issues regularly all over the world on a daily basis - some of them are small and some of them are large. In July issues at the FAA New York area caused hundreds of flights to be cancelled and thousands delayed. Barcelona Tower had a power cut in August for only a few minutes and this led to 10 diversions, 5 cancellations and dozens of other flights incurred huge delays. These are just two recent examples. Italian ATC have two strikes planned in the near future. Two years ago Edinburgh ATC (non NATS) suffered huge delays one afternoon. You will only hear about these delays when you are affected by them or they make national news headlines.
The reliability of ATC is not just measured in terms of how frequently the system breaks due to errors in technology. Reliability of ATC is measured in how well they cope during bad weather, ground disruptions, airline disruption and staff shortages (strikes etc), emergency traffic and so on. NATS is very capable in coping with all of the above whereas many other control units around world are not. I believe NATS sets the standard on Air Traffic Control both in its levels of operational capacity, capability and reliability.
That is not to say that what occurred is acceptable - it absolutely isn’t. But the level of disruption caused by NATS failures is a direct consequence of how busy the skies and airports are in the UK.
NATS clearly do have an incentive to provide a reliable service. If they don’t then serious incidents happen. There is also the professional pride of the controllers.
These type of disruption events show how important it is to buy comprehensive travel insurance as soon as you book a trip. I’d say that goes for any mode of transportation. ‘Compensation’ is highly unlikely to be forthcoming due to the NATS disruption but with a solid insurance policy the holder should at least not be out of pocket.