Nats fallen over again

[Tetragon213]

Most won't believe the many, many attacks on infrastructure were Russia unless there is a verified signed document by Putin saying "I did it - Yours, Putin".

edit: Crazy how many people refuse to accept we're in a hybrid war with Russia and have been for a long time. Yes I know there are other causes for things

Oh, I can do you worse.

Had a housemate who outright defended Russia attacking us, with some choice words that boiled down to justifying Russia attacking other nations.

Some people (we would call them Tankies) are genuinely beyond hope.

 

[Edsmith]

There is still a Right to Care in place. If there's enough stock to give out.

That's it, if there is enough stock to give out. The airlines have had the rug pulled from under them and must be in a state of disarray.

 

[nlogax]

Interesting to read that Martin Rolfe still commands the confidence of No.10 to at least report back on the outcome of the investigation into this week's mess. My guess is once he's shared the root cause and the investigation's findings he'll cease to be CEO of NATS.

https://www.standard.co.uk/news/uk/...f-executive-air-traffic-control-b1296285.html

“[Heidi Alexander] asked him to carry out a full investigation into what happened to make sure that lessons are learned, so that this doesn’t happen again.
“She set a deadline for them to report back in a week’s time.”
Asked whether Ms Alexander and Andy Burnham still have confidence in Mr Rolfe, the Prime Minister’s spokesman added: “It’s self-evident that she wants answers and the full investigation to be reported back in a week’s time, and she’s got confidence in him to deliver that.”

This statement appeared to fall short of full confidence in Mr Rolfe.
Pressed whether this meant the Government’s confidence in the Nats’ chief was conditional on the outcome of the investigation, the spokesman replied: “We have confidence in him and he’s been tasked with the investigation, which will report in a week’s time.”

 

[Bantamzen]

Interesting to read that Martin Rolfe still commands the confidence of No.10 to at least report back on the outcome of the investigation into this week's mess. My guess is once he's shared the root cause and the investigation's findings he'll cease to be CEO of NATS.

https://www.standard.co.uk/news/uk/...f-executive-air-traffic-control-b1296285.html

After 3 major fails in 3 years, he should be packed off to the job centre. If most ordinary folk where at the heart of such a monumental cock up* we'd be out on our ear. But this is UK plc, where the consumer doesn't matter so long as the profits roll...

(*I know that the suspicion is a cyber attack from some quarters, but even if it was this there should have been better protection / mitigation in place for such a critical system)

I got caught up in this yesterday trying to fly out to Crete from Manchester. The really galling thing was when we finally found out we were being cancelled and the holiday binned, I could literally see the craft allocated to us on one of the remote stands. Worse still was the complete breakdown in comms, I know that staff on the ground knew little more than us, but when the time came for cancelling multiple Jet2 flights hundreds of use were just walked out to Border Control without anything more than "it's been cancelled" from staff. Were we to wait landside for a later flight, or one in the morning? Were our holidays canned? Nothing was clear until emails started to arrive confirming that we had our holidays cancelled and will eventually get a refund. This was around 9pm last night, in the baggage hall with thousands of increasingly angry passengers trying to find out if they'd even get their bags, let alone which carousel scrum you had to go to. Thankfully it "only" took 2 hours from leaving departures for me to get my suitcase, the hall was still rammed when I shoulder barged myself to the exit.

I've already ranted about this on social media, so I'll keep this short. It is time to bring key infrastructure systems like NATS back into full public control, and more importantly accountability. One failure is a failure too many, 3 is criminal. And more importantly, situations like this should trigger compensation not just refunds for cancellations. It's an absolute disgrace that some people have to wait for days to get flights home as some poor souls are having to do, or just simply having to mill about increasingly packed and stressful areas like the baggage hall whilst desperately trying to work out which belt their luggage may appear. Perhaps if such measures were in place, some minds might focus a bit more.

 

[renegademaster]

The problem with going on too much about foreign adversaries is ultimately there's not actually a huge amount you can actually do about them other than not have vulnerabilities for them to exploit in the first place, which is prevented by good quality control and security practices that are a good idea regardless.

 

[Stossgebet]

Even if it was malicious action, Russia being a top 3 suspect if so. People seem to forget that our government wouldn't be shouting about it. For evidence, see how a (potentially made up) Russian intermediary was blamed for ordering the firebombing of then Prime Ministers Keir Starmers property. And the during the court case, the judge specifically ruled out questioning the underlying motivations behind the firebombing/arson attacks. And the whole shameful scandal was barely news.

 

[Sun Chariot]

DDOS cyber attacks are just as likely to come from nations other than Russia.
I was at the organisation when a DDOS attack occurred. The identification and preventative action happened within a very short space of time: seconds. Contingency operations kicked in within minutes.

August 2024's outage was due to, iirc, bad data within a daily routine interface from France.
So I am more inclined to think this will be a root cause where a correct failover was not identified or acted upon in a timely manner; thus compounding and lengthening its effect.

Whatever the findings, I hope the operation emerges more resilient. I also hope that the Public-Private* structure aids, not hinders, the decisions or investments recommended.

* UK Government owns 49% of NATS; and it holds a "golden share" with special voting rights to block decisions—such as protect national security interests.
The Airline Group (a consortium of airlines) owns 42% NATS.
London Heathrow Airport Holdings owns 4%.

 

[Bald Rick]

I got caught up in this yesterday trying to fly out to Crete from Manchester. The really galling thing was when we finally found out we were being cancelled and the holiday binned, I could literally see the craft allocated to us on one of the remote stands. Worse still was the complete breakdown in comms, I know that staff on the ground knew little more than us, but when the time came for cancelling multiple Jet2 flights hundreds of use were just walked out to Border Control without anything more than "it's been cancelled" from staff. Were we to wait landside for a later flight, or one in the morning? Were our holidays canned? Nothing was clear until emails started to arrive confirming that we had our holidays cancelled and will eventually get a refund. This was around 9pm last night, in the baggage hall with thousands of increasingly angry passengers trying to find out if they'd even get their bags, let alone which carousel scrum you had to go to. Thankfully it "only" took 2 hours from leaving departures for me to get my suitcase, the hall was still rammed when I shoulder barged myself to the exit.

Sounds like your issue is more with MAG and Jet 2. Perhaps it’s their CEOs who should be hung out to dry.

 

[Sun Chariot]

Sounds like your issue is more with MAG and Jet 2. Perhaps it’s their CEOs who should be hung out to dry.

+1 to this.

NATS does air traffic management in Upper Airspace and in Lower Airspace and for some UK airports' Terminal Control.

 

[Tetragon213]

The problem with going on too much about foreign adversaries is ultimately there's not actually a huge amount you can actually do about them other than not have vulnerabilities for them to exploit in the first place, which is prevented by good quality control and security practices that are a good idea regardless.

If someone is determined enough, and has sufficient resources, they will find a way through even the most elaborate of defences. It has been noted that the timing is rather odd, as it comes just hours after sanctions were placed on West Bank settlements; this has led to some humorous (and some less light-hearted) discourse about "Mossad hackers"; see the FlightRadar24 Twitter post...

 

[BananaRepublic]

If someone is determined enough, and has sufficient resources, they will find a way through even the most elaborate of defences…….

They can’t find a way through where it’s not connected to any publicly, or wider www networks.
In other words, they cannot reach which is unreachable.

The cause of this outage is said to be known and has been fixed.
The NATS systems are a combination of very complex individual systems working together.
The FDP (flight data) element is crucial, as it “feeds”, underpins and interacts with the other systems.
The levels of redundancy are far higher and more resilient than almost anything on the planet, but as with any massively complex environments, there will be latent weaknesses, particularly when dealing with FDP.

The previous significant outage occurred due to a subsidiary system (basically an interpreter) falling over in a million to one scenario (bad flight data).
We will be told where this issue occurred in due course.

No change of management or oversight can alter the fact that the NATS system is a very complex and highly advanced beast.
The radar tools and controller workstation tech is class leading, however the underlying FDP system is ancient, but very capable and usually highly dependable.
They’ve been working on a replacement FDP, along with European colleagues, for more than a decade, but it is has proved to be a very difficult process.

Failures like this may be catastrophic for airline operations and for passengers, but they occur safely and the operation is slowed down, or closed down in a controlled manner.
Unfortunately the knock on effects go on for a lot longer than the hour or two that contingency operations are in force.

The skies are so busy, that when the highly complex ATC systems have failures or outages, it’s impossible to maintain the normal level of air traffic movements. Fact.

 

[renegademaster]

If someone is determined enough, and has sufficient resources, they will find a way through even the most elaborate of defences

If this was true, they'd also hack something more consequential like Visa Or MasterCard

 

[Watershed]

The previous significant outage occurred due to a subsidiary system (basically an interpreter) falling over in a million to one scenario (bad flight data).

The flight plan was valid, the system just couldn't cope with it. Given the number of flights that take place each year, "one in 15 million" (which is the figure that NATS gave as an excuse) really means "every 10 years". I don't think that's a remotely acceptable level of reliability for such a crucial piece of national infrastructure

The levels of redundancy are far higher and more resilient than almost anything on the planet, but as with any massively complex environments, there will be latent weaknesses, particularly when dealing with FDP.

It's clearly not very redundant or resilient at all if it can go down like this for such a long time.

We already saw with the flight planning interpreter failure that if a particular flight plan triggers a bug in the code, the backup system could lock up/fail in just the same way because the systems aren't designed to skip the problematic flight plan for manual intervention, and continue working otherwise.

I wouldn't be surprised if it's a case of a similar type of failure in terms of poor systems design.

 

[mpthomson]

The obvious possibility here is that a certain vodka drinking nation might have been busy...

Ryanair has said that its a repeat of the issue in 2023 (and that was nothing whatever to do with Russia, or Israel, which is the latest theory from the crazies, in response to Milliband's announcement).

 

[pitdiver]

After 3 major fails in 3 years, he should be packed off to the job centre. If most ordinary folk where at the heart of such a monumental cock up* we'd be out on our ear. But this is UK plc, where the consumer doesn't matter so long as the profits roll...

(*I know that the suspicion is a cyber attack from some quarters, but even if it was this there should have been better protection / mitigation in place for such a critical system)

I got caught up in this yesterday trying to fly out to Crete from Manchester. The really galling thing was when we finally found out we were being cancelled and the holiday binned, I could literally see the craft allocated to us on one of the remote stands. Worse still was the complete breakdown in comms, I know that staff on the ground knew little more than us, but when the time came for cancelling multiple Jet2 flights hundreds of use were just walked out to Border Control without anything more than "it's been cancelled" from staff. Were we to wait landside for a later flight, or one in the morning? Were our holidays canned? Nothing was clear until emails started to arrive confirming that we had our holidays cancelled and will eventually get a refund. This was around 9pm last night, in the baggage hall with thousands of increasingly angry passengers trying to find out if they'd even get their bags, let alone which carousel scrum you had to go to. Thankfully it "only" took 2 hours from leaving departures for me to get my suitcase, the hall was still rammed when I shoulder barged myself to the exit.

I've already ranted about this on social media, so I'll keep this short. It is time to bring key infrastructure systems like NATS back into full public control, and more importantly accountability. One failure is a failure too many, 3 is criminal. And more importantly, situations like this should trigger compensation not just refunds for cancellations. It's an absolute disgrace that some people have to wait for days to get flights home as some poor souls are having to do, or just simply having to mill about increasingly packed and stressful areas like the baggage hall whilst desperately trying to work out which belt their luggage may appear. Perhaps if such measures were in place, some minds might focus a bit more.

My wife and I were in exactly the same position. We were due to fly out from East Midlands Airport at 15.05 to Fuerventura for a special holiday. We were sat on the aircraft for two hours but eventually told to return to the terminal. Where we sat for another couple of hours. Then someone noticed that our flight had disappeared from the screens. Eventually and I mean eventually we were advised the flight had been cancelled and all associated holidays as well. All we were told was we would get a refund. Then the emails and texts followed. Forgot to mention it was with Jet2.

 

[Terry Tibbs]

‘I've already ranted about this on social media, so I'll keep this short. It is time to bring key infrastructure systems like NATS back into full public control, and more importantly accountability.’
I struggle to understand the logic to first part of the quote. I do not see how or why NATS being in public control would have possibly prevented its systems falling over. I agree with full accountability and leadership taking the responsibility for these messes though. However bringing NATS entirely under public control I fear would make it as effective, efficient, dynamic and flexible as other state controlled monoliths. The levels of accountability in government run organisations is also as close to zero as possible.
UK ATC is extremely good and the level of control is world class. UK airspace (London in particular) is some of the busiest and complex in the world. I am certainly not saying it’s perfect and incidents like what has just happened should be expected - they shouldn’t. But I for one would not want NATS to be owned by the state. Certainly government oversight and regulation but not public ownership.

 

[Sun Chariot]

I agree @TerryTibbs and, in post 37, @Bantamzen , I laid out the primary owners - and influencers - of NATS:
49% UK Government, which also has power to veto decisions of other owning entites.
42% The Airline Group, a consortium of the airlines operating into and out of the UK.
4% London Heathrow Airport Holdings.

Those latter two entities - for airlines and airports - act in part as a "conscience". I think that conscience risks being lost - or at least diminished - in a fully state-owned org.