All cross country services suspended 23/08/26

[Amlag]

I trust the Government will now initiate a full inquiry into this XC Fiasco including any involvement by ‘outside of the UK organisations etc.’

 

[NCT]

Putting a tinfoil hat on for a moment, it wouldn't surprise me if the full extent of the event is only revealed once papers are released in 30 years' time.

 

[Horizon22]

This is something Thameslink often do when there is major disruption to their services. They remove the cancelled services from the departure boards, to only show services that are confirmed as running. Granted they do also explain this with an additional message on the departure boards.

Yes it’s called “Disruption Mode” a common feature on many CIS platforms.

Paying for a second office location forever vs a once-in-thirty-year shut-down event ...

Many BUCFs (back up control facilities) are significantly smaller and shoehorned into places like crew depots, head offices and other spare railway land and could run with skeleton resources and technology and might be shook down once a year to test out the contingency.

As others have said XC have used the same building for decades so it is not necessarily “negligence” or “heads such roll” for using such a shared facility, especially when it would have been some time since such a decision was made.

The more awkward questions will be why they couldn’t then operate practically at all without access. If the company suffered some sort of critical IT failure, then the physical location may not have mattered if people couldn’t access details even their cloud or VPN site. Also many telephony systems can’t be transferred to just run from a mobile as they are bespoke. But the fact remains that if this was a physical issue (let’s say Cannon House had a gas leak outside with no access), would the outcome have been any different?

 

[norbitonflyer]

That wouldn't work for me at all during the approximately four hours it was up for while I was trying to do something with it. Typically the failure was when I did want to use my account to make a payment.

This is more acute with Monzo because they don't provide any in-person or telephone services. Also from some journalists and tweets a number of other users were also unable to make any payments. I was able to view stale balances but not receive or make payments. I'm considering complaining actually, as it could have been very bad. Luckily I avoided economic losses.

Monzo isn't the only one. The bank used by the charity of which i am the treasurer has had several outages over the summer (including right now) and is currently only fully available online during normal banking hours (which is a real pain if you are a volunteer with an unrelated day job). This follows a badly botched relaunch of their website last year.

 

[liamf656]

So does anyone actually know what happened, also how come Peterborough to Stansted continued to operate (short of declaring UDI or defecting to Greater Anglia)_

I think it's because it's self contained today due to engineering works, so the stock and staff arrangements would be very easy to manage, as opposed to the rest of the (very complex) network

 

[Bletchleyite]

Not everyone has the benefit of VFH and many places still insist on Office working with deskbound PC's, and for many companies resiliance/duplication is a dream.

Those businesses are therefore somewhere between incompetent and grossly negligent. Their shareholders would have a good case against them for the loss of value or dividend the resulting outages would cause.

As for resiliance what do you do when the cloud takes a nap, of Microsoft Teams (if you use it) takes a lie down (taking out phones as well)? Many companies reply on these so much now (along with AI but lets steer clear of that rabbit hole) and when they fail everything stops. At my last job before retiring the mantra was Cloud! Cloud! Cloud! (before coming AI! AI!), no desk phones (all Teams), even email in the cloud,

Cloud services are very resilient, but even then you have workarounds in place. A Teams outage won't usually take email out, and a directory of mobile numbers provides an added defence.

I do wonder what they do when it all falls over or a JCB takes out the lines to a manufacturing plant (you can't take an assembly line home with you)

Manufacturing isn't services. You can't compare the two.

Sometimes with the way everythings is getting interconnected I wonder that we have so few outages

Because most businesses, unlike XC, have proper resilience plans in place.

 

[Dave61]

As others have said XC have used the same building for decades so it is not necessarily “negligence” or “heads such roll” for using such a shared facility, especially when it would have been some time since such a decision was made.

The more awkward questions will be why they couldn’t then operate practically at all without access. If the company suffered some sort of critical IT failure, then the physical location may not have mattered if people couldn’t access details even their cloud or VPN site. Also many telephony systems can’t be transferred to just run from a mobile as they are bespoke. But the fact remains that if this was a physical issue (let’s say Cannon House had a gas leak outside with no access), would the outcome have been any different?

From what I have read it was a power outage = no computers running. For all we know their IT room could have been a small room off to one side with a small UPS for power blips and and air-conditioner (we don't actually know).
but with no power in the building there were no (or it seems like it) computers/servers to connect to.
For the gas leak the power may have stayed on and the computers running.....

 

[Towers]

I trust the Government will now initiate a full inquiry into this XC Fiasco including any involvement by ‘outside of the UK organisations etc.’

I would suggest that the Government are rather unlikely to admit it if a rogue nation had managed to halt the railway! It will forever remain a ‘power cut’!

== Doublepost prevention - post automatically merged: Sunday at 22:51 ==

Consequences for whom?

XC to DfT - we need to spend £1m on improved Business Continuity Plan/infrastructure. Dft to XC - Er, no. .....

This.

It is always worth reminding folk, when the complaints roll in as they do, that XC are the abysmal operation that they are very much at the DfT’s behest. Failure to ensure that they got the full quota of off lease Voyagers despite the legendary, decades long routine overcrowding being a particularly notable recent example of the prevailing utter contempt for XC’s passengers that appears to be maintained by Whitehall.

 

[Bletchleyite]

From what I have read it was a power outage = no computers running. For all we know their IT room could have been a small room off to one side with a small UPS for power blips and and air-conditioner (we don't actually know).

If that was their sole IT facility then it is not fit for purpose. If you're running operationally critical stuff onsite for a large business you really need at least two independent facilities.

If you're using cloud facilities your cloud provider has that (many times over).

 

[Dave61]

Those businesses are therefore somewhere between incompetent and grossly negligent. Their shareholders would have a good case against them for the loss of value or dividend the resulting outages would cause.

I think you must be living in the Green and Sunny Uplands, DR/BC can cost serious money and not every company can afford it, there are thousands of companies out there, some medium/small that may not even have an IT department.
I assume your company has a DR/BC plan for a comet strike vapourising central London ? Yes it is a deliberatly absurd suggestion but it is a valid senario to consider, in this case the odds would be astronomical (sorry for the pun), this costs of implimenting it literally out of this world (DC on the moon?) and your customers would probably be vapourised as well anyway.
The point I am trying to make (repeatably) is that every senario has a cost/benefit ratio, and here we have a single event in 31 years for XC (the recent ROC's are another story), so someone decided it was not worth it. There is no point implimenting something if it bankrupts the company

 

[Starmill]

If that was their sole IT facility then it is not fit for purpose. If you're running operationally critical stuff onsite for a large business you really need at least two independent facilities.

If you're using cloud facilities your cloud provider has that (many times over).

A lot of the software isn't supplied in a cloud-based format. Of course anything is possible but every operator has a slightly different combination of control software and interfaces. To achieve what you're suggesting I'd suggest a large in-house development project would probably be needed as there'd be no single external supplier (Magellan, Siemens, IVU Traffic Technologies, whoever) willing to take on such a risky migration. Tens of millions to build the system from scratch essentially.

Now I'd say that's a good use of money. But ultimately the industry probably wouldn't as it's so much cheaper to continue with the current cobbled-together mess.

 

[Dave61]

If you're using cloud facilities your cloud provider has that (many times over).

Only if you configure and pay for it, you will be surprised to know that this is not automatically the case (ask the customers of OVHcloud and it's SBG2 data centre in Strasbourg)

 

[Bletchleyite]

I think you must be living in the Green and Sunny Uplands, DR/BC can cost serious money and not every company can afford it, there are thousands of companies out there, some medium/small that may not even have an IT department.

XC is a large business. Smaller businesses are best going towards cloud as those providers do that for you.

I assume your company has a DR/BC plan for a comet strike vapourising central London ?

Yes, the product I work on runs from multiple data centres worldwide and staff in multiple locations too.

Yes it is a deliberatly absurd suggestion

It's not.

but it is a valid senario to consider, in this case the odds would be astronomical (sorry for the pun), this costs of implimenting it literally out of this world (DC on the moon?) and your customers would probably be vapourised as well anyway.

Nope.

The point I am trying to make (repeatably) is that every senario has a cost/benefit ratio, and here we have a single event in 31 years for XC (the recent ROC's are another story), so someone decided it was not worth it. There is no point implimenting something if it bankrupts the company

A business critical single point of failure like this is incompetent. Whataboutery does not change that.

 

[IanXC]

According to a reply by Xcountry on Facebook just before today's shambles, they are experiencing significant delays in processing delay repay claims. So even getting a refund could take weeks. At what point does the government take the contract back? Trains are still regularly cancelled because of a lack of drivers.

If you want a refund this has nothing to do with CrossCountry's Delay Repay processing?

== Doublepost prevention - post automatically merged: Sunday at 23:36 ==

That is pathetic, but nothing about the abysmal information system on our railways today surprises me anymore.

So responding to passenger feedback via the NPS is pathetic. Oh interesting.

 

[josh-j]

A business critical single point of failure like this is incompetent. Whataboutery does not change that.

All sorts of large businesses have had embarrassing major outages. Remember when Facebook locked themselves out of their datacentre?

I think it's hyperbole to call this incompetence without knowing what actually happened. The fact they crashed and burned for the best part of a day doesn't imply incompetence, sometimes even if you're one of the biggest companies in the world you can still mess up your multi-billion dollar network worldwide, because in reality s*** does happen occasionally.

 

[class 9]

How many other significant issues has it caused that have been detrimental to train operation since XC first started using Cannon House 20 years ago?

In the last few years, numerous IT and telephone line failures.

 

[45Fox]

I am booked to travel from Leicester to Newport on Monday on two Cross Country services.
There are quite a few other Glamorgan cricket fans returning from here to South Wales tomorrow
Hope the trains are running tomorrow morning.

The currently timetabled services (subject to planned engineering work) between Leicester and Birmingham ran in full from 18.00 until the end of services on Sunday evening, there's a full complement of XC 170s on Leicester depot tonight so the omens for trains to Birmingham on Monday are good; I wouldn't like to comment on the prospect of services from New Street to Newport though.

Commiserations on losing at Grace Road, although it was a great result for us Foxes, consecutive victories in Division One of the County Championship? I can't remember the last time!

Wishing you all a safe trip back home tomorrow.

 

[Kettledrum]

Where was the contingency planning or IT disaster recovery plan here? If there was a plan, then when was it last tested to make sure that it could be implemented immediately?

It looks like XC had a single point of failure and was incapable of putting emergency alternative plans into action, which is really poor.

 

[Bow Fell]

There was a contingency, which has been mentioned already and had been in operation which would have gone unnoticed, had it not been for the IT systems/phones in the “contingency control” failing, so a third contingency required?!

 

[Djgr]

Was BR just as brittle but it's just with poorer communications things like this happened without people generally being aware?

== Doublepost prevention - post automatically merged: Sunday at 20:40 ==

No

In many ways communication pre internet was more effective as newspapers and television reached a high percentage of the population.

 

[Harpo]

A business critical single point of failure like this is incompetent.

A single point of failure? Or a concatonation? We are still without any information at all on immediate or underlying causes but have no end of eye-wateringly expensive ‘solutions’ on offer such as duplicate locations.

Whataboutery does not change that.

Everything here about causes is currently whataboutery.

 

[GoneSouth]

No

In many ways communication pre internet was more effective as newspapers and television reached a high percentage of the population.

And what was out there for everyone to see had generally been subject to some rigour and journalistic professionalism rather than every man and his dog throwing muck at Twitter.

Hmmm, well on the other hand, there was The Sun as well i suppose!

Happy days

 

[Furryanimal]

The currently timetabled services (subject to planned engineering work) between Leicester and Birmingham ran in full from 18.00 until the end of services on Sunday evening, there's a full complement of XC 170s on Leicester depot tonight so the omens for trains to Birmingham on Monday are good; I wouldn't like to comment on the prospect of services from New Street to Newport though.

Commiserations on losing at Grace Road, although it was a great result for us Foxes, consecutive victories in Division One of the County Championship? I can't remember the last time!

Wishing you all a safe trip back home tomorrow.

Thanks..it was a good game though.

 

[AdamWW]

In many ways communication pre internet was more effective as newspapers and television reached a high percentage of the population.

Perhaps but the threshold for something being worth reporting is somewhat higher than for someone posting in a web forum.

 

[Djgr]

Perhaps but the threshold for something being worth reporting is somewhat higher than for someone posting in a web forum.

Pretty certain that thousands of people stranded because of no trains would have been reported!

 

[Russel]

In the last few years, numerous IT and telephone line failures.

That wasn't what I asked, my question was how many other failures as a result of using Cannon House have had a direct impact on train operations?

This is the first time I can recall where their has been wide spread disruption.

Not trying to defend XC, just trying to put things into perspective.

 

[AlterEgo]

One sector of BR being unable to function would have been less big a deal back in the day. On a national scale only a small fraction of travellers were delayed or affected by yesterday's events, and, importantly, it didn't affect London. Amusing to read the paranoia around whether the Iranians or Russians did it to be honest. Why would they pick *CrossCountry*?

 

[DarloRich]

A couple of points to consider:

• We don't know what has happened, why or when
• Sometimes, despite the best laid plans excrement happens!
• It is a perfectly fair business decision to say that the risks of complete service loss are so small that we wont spend any money on continuity and will carry the risk of failure and deal with any reputational damage.
• Not everything is a conspiracy. Almost all of the time it is a simply a cock up
• Step away from the Tom Clancy novels. I doubt the Russians are behind this. Comrades we have crippled the economic performance of perfidious Albion. Great! have you crashed the banking system and stock market? NO. We have caused XC to be unable to run trains...............

It is also perfectly possible no one thought about this happening!

 

[Skymonster]

Will there be any consequences for XC for being unable to run a service because of poor business decisions?

Consequences for whom?

XC to DfT - we need to spend £1m on improved Business Continuity Plan/infrastructure. Dft to XC - Er, no. .....

This… For some large businesses, failing to deliver a service to its customers could cause significant reputational and financial damage. But for Cross Country? Its reputation is already in the gutter due to overcrowding, cancellations, and strikes. It’ll have to refund some customers because they didn’t travel yesterday but many will just have self-transferred to services operated by other companies. The outage will have no major consequences because the vast majority of people who need to use Cross Country train services will use them again despite what’s happened. So the harsh reality is that there’s no incentive to make the business truly resiliant.

The really sad thing is not that Cross Country wasn’t resiliant. Its that vehicles operated by humans are unable to be used without the life-line of being able to contact a remote control centre - and apparently without being able to get in touch via a land-line at the control centre end. How did the railway run before computers and mobile phones I wonder?

 

[TrainTraveler]

A couple of observations on this

1) A bit like the Heathrow power issue of a while back, taking the "brave" decision of no trains on the day to stand a good chance of a "normal" train service the following day might have been the best choice.

2) I'm sure that consideration will be given to shared "control" facilities, so each TOC swaps half of their "control" facility with another so that a degraded "control" is available even if there is a bulding & systems failure. This will be much easier under a common (GBR) ownership.

A two site model would seem sensible but it's not a trivial task to change systems that have been single site to working well between two sites. A true "DR" facility is hard to keep up to a satisfactory standard and the split site may be the best approach for the future.