All cross country services suspended 23/08/26

[Tetchytyke]

Amusing to read the paranoia around whether the Iranians or Russians did it to be honest. Why would they pick *CrossCountry*?

The Russian, Iranian, and North Korean hacking groups (or, indeed, kids in their bedrooms in the West Midlands) don’t ever really choose their target, they tend to work on a strategy of flinging muck around the Internet and finding out what sticks. Big companies like TOCs are often more vulnerable because they still use legacy software on legacy systems.

 

[DarloRich]

A two site model would seem sensible but it's not a trivial task to change systems that have been single site to working well between two sites. A true "DR" facility is hard to keep up to a satisfactory standard and the split site may be the best approach for the future.

First question when times are even the slightest touch hard: why are we paying for this extra site we never use?

 

[cats_five]

A couple of points to consider:

• We don't know what has happened, why or when
• Sometimes, despite the best laid plans excrement happens!
• It is a perfectly fair business decision to say that the risks of complete service loss are so small that we wont spend any money on continuity and will carry the risk of failure and deal with any reputational damage.
• Not everything is a conspiracy. Almost all of the time it is a simply a cock up
• Step away from the Tom Clancy novels. I doubt the Russians are behind this. Comrades we have crippled the economic performance of perfidious Albion. Great! have you crashed the banking system and stock market? NO. We have caused XC to be unable to run trains...............

It is also perfectly possible no one thought about this happening!

Thanks for the giggle at your last point! Agree with all you said. That was my experience of working in IT for most of my life.

 

[AlterEgo]

The Russian, Iranian, and North Korean hacking groups (or, indeed, kids in their bedrooms in the West Midlands) don’t ever really choose their target, they tend to work on a strategy of flinging muck around the Internet and finding out what sticks. Big companies like TOCs are often more vulnerable because they still use legacy software on legacy systems.

There hasn't been a hack though.

 

[Harpo]

There hasn't been a hack though.

Sources?

 

[DarloRich]

Thanks for the giggle at your last point! Agree with all you said. That was my experience of working in IT for most of my life.

I suspect this will be part of the answer:

There was an important job to be done and Everybody was sure that Somebody would do it. Anybody could have done it, but Nobody did it. Somebody got angry about that because it was Everybody’s job. Everybody thought Anybody could do it, but Nobody realized that Everybody couldn’t do it. It ended up that Everybody blamed Somebody when Nobody did what Anybody could have.

 

[cats_five]

I suspect this will be part of the answer:

There was an important job to be done and Everybody was sure that Somebody would do it. Anybody could have done it, but Nobody did it. Somebody got angry about that because it was Everybody’s job. Everybody thought Anybody could do it, but Nobody realized that Everybody couldn’t do it. It ended up that Everybody blamed Somebody when Nobody did what Anybody could have.

My experience suggests no-one realised that job had to be done. Or, as others have suggested, XC looked at the cost of doing it and decided to take the risk.

 

[Tetchytyke]

If it does really turn out that there were no workable back-up systems in place, the boss of Cross Country should fall on her sword.

Quite ironic really that she used to work for Amazon. Maybe they could do her a deal on cloud-based disaster recovery.

There hasn't been a hack though.

I don’t know if there has or hasn’t, but the answer was to the question “why choose XC?”

That said, the Co-Op, M&S, and TfL all said that they’d not been hacked and it was just an outage.

 

[AlterEgo]

Sources?

Oh I don't give away my sources but it may help to know that I used to work there.

== Doublepost prevention - post automatically merged: Monday at 08:43 ==

I don’t know if there has or hasn’t, but the answer was to the question “why choose XC?”

That said, the Co-Op, M&S, and TfL all said that they’d not been hacked and it was just an outage.

How many of them said this was because of a power cut meaning they couldn't get into their building on a Sunday yet they are all magically back to work on a Monday?

 

[Haywain]

Or, as others have suggested, XC looked at the cost of doing it and decided to take the risk.

And any such decision would be made with a close eye on the remaining term of the franchise, which defines the period of risk.

 

[SuspectUsual]

And any such decision would be made with a close eye on the remaining term of the franchise, which defines the period of risk.

Which, if that is the case, reflects poorly on the person letting the franchise.

 

[AndrewE]

And any such decision would be made with a close eye on the remaining term of the franchise, which defines the period of risk.

shouldn't that decision have been made at the start? i.e. the risk to the business which started from day 1, or the first day they moved into that building.

 

[TrainTraveler]

DarloRich wondered

"First question when times are even the slightest touch hard: why are we paying for this extra site we never use?"

It's a very valid question.

As others have observed, in a strict financial sense, split sites (rather than a duplicated DR site) may not be justified. You do, however, need to consider the "servica availability" and "reputational risk" aspects.

If implemented as part of other changes (for instance bringing TOCs into GBR or changes to the IT systems that facilitate the railway operating) site splits can be implemented at modest cost.

At the end of the day, someone has to make the call as to whether single site operation is good enough. That person occasionally gets to justify that choice.

I wouldn't be surprised if that were playing out this morning....

 

[Russel]

• Step away from the Tom Clancy novels. I doubt the Russians are behind this. Comrades we have crippled the economic performance of perfidious Albion. Great! have you crashed the banking system and stock market? NO. We have caused XC to be unable to run trains...............

Please don't give XC ideas for new excuses to cancel trains!

 

[Haywain]

shouldn't that decision have been made at the start? i.e. the risk to the business which started from day 1, or the first day they moved into that building.

Even then it would have been a term of only 7 years or so.

 

[D6130]

yesterday's events, and, importantly, it didn't affect London. Amusing to read the paranoia around whether the Iranians or Russians did it to be honest. Why would they pick *CrossCountry*?

No paranoia on my part....

It's only a couple of days since Comrade Putin said that the UK would pay for continuing to supply weapons and drones to Ukraine.

....the emoji is a hint that it was merely a humorous suggestion!

By the way, are Cross Country back to running a 'normal' service this morning?

 

[sh24]

Having written business continuity and disaster recovery plans, and sadly had to act on them more than once, there are ALWAYS curveballs, wrinkles or unexpected stuff that happens. DR and BC plans are intended to help and guide you respond to a situation. Beyond the immediate danger to life and limb (e.g. fire, and very memorably for me the Buncefield oil depot explosion) they don't provide a perfect check list of actions that will result in your business or operations magically regenerating in another location.

 

[Dave61]

DarloRich wondered

"First question when times are even the slightest touch hard: why are we paying for this extra site we never use?"

It's a very valid question.

As others have observed, in a strict financial sense, split sites (rather than a duplicated DR site) may not be justified. You do, however, need to consider the "servica availability" and "reputational risk" aspects.

If implemented as part of other changes (for instance bringing TOCs into GBR or changes to the IT systems that facilitate the railway operating) site splits can be implemented at modest cost.

The transport secretary was strongly hinting that going forward with GBR and it would make sense then for 3 or 4 control centres spread around the country with the capacity to pick things up if one (or more) fails, it will also be much easier under the roof of a single entity called GBR
There will be challenges however as it may mean closing some/moving people around (which can cause issues - sure close some but not "mine") and will cost some money to set up, train people and test.
And we all know how government IT projects go...often badly.

== Doublepost prevention - post automatically merged: Monday at 09:56 ==

Thanks for the giggle at your last point! Agree with all you said. That was my experience of working in IT for most of my life.

Which is what I have been trying to get across in my posts, also my experience of working in IT for most of my life. We all don't have £billions to throw at the problem.

 

[apinnard]

Only if you configure and pay for it, you will be surprised to know that this is not automatically the case (ask the customers of OVHcloud and it's SBG2 data centre in Strasbourg)

Or most of the globe when CloudFlare went t**s up at least once last year. Sure, you can host your dns with a big provider, but nobody is totally infallible.

 

[Dave61]

This… For some large businesses, failing to deliver a service to its customers could cause significant reputational and financial damage. But for Cross Country? Its reputation is already in the gutter due to overcrowding, cancellations, and strikes.

And who else are you going to travel with ? On a lot of routes XC is the only choice

 

[Bletchleyite]

And who else are you going to travel with ? On a lot of routes XC is the only choice

Relatively few journeys cannot be made avoiding XC (though doing so would usually add a change of trains). If this had happened to LNER, say, the effect would be far worse.

 

[apinnard]

The point I am trying to make (repeatably) is that every senario has a cost/benefit ratio, and here we have a single event in 31 years for XC (the recent ROC's are another story), so someone decided it was not worth it. There is no point implimenting something if it bankrupts the company

Agree. I own a small MSP in the Midlands. One of my clients who is a large design and manufacturing house for white labelled clothing that sells into supermarkets and low-cost fashion chains recently instructed my company to remove their DRaaS replica virtual infrastructure in a data centre to cut costs.

They are running on bare metal backups stored on devices in their single building now, and that’s it. They accepted the risk (in writing, and indemnified us). We are seeing that sort of discussion a lot more as profits are hit with small and medium businesses.

 

[81A]

Even then it would have been a term of only 7 years or so.

And if this is a legacy systems issue, the risk of something going wrong probably increases gradually (after all, an entire system based on Windows XP would have been cutting edge in 200x, by 201x a bit dated but not crazy and by 202x becomes ridiculously insecure/unreliable).

 

[Djgr]

Let's hope it wasn't caused by a fax machine overheating...

 

[Rail Ranger]

It's a normal service today on Cross Country. Just 13 cancellations: Service Updates | CrossCountry Trains https://share.google/EyGWwvvyq3FUwHCPCService Updates | CrossCountry Trains https://share.google/EyGWwvvyq3FUwHCPC

 

[zwk500]

First question when times are even the slightest touch hard: why are we paying for this extra site we never use?

Worth pointing out that in Birmingham there is already a Network Rail Control Centre where XC would not need to do more than agree for a couple of desks to be set aside for them. Given how critical a hub Birmingham New Street, using that control centre on a regular basis would also make a lot of sense. It'd also mean that big things like access to systems and silly things like staff access cards would be already set up for emergency relocation.

Even more handily, Birmingham ROC is a half-hour walk, 10 minute taxi, or 10 minute bus ride from XC's offices.

 

[py_megapixel]

Relatively few journeys cannot be made avoiding XC (though doing so would usually add a change of trains).

There still a fairly significant axis (Derby - Birmingham - Cheltenham) on which they are the only option.

You can just about avoid the Birmingham-Cheltenham bit by going round via Worcester, but it takes three times as long.
From Derby you can get to Stoke - but again, very slowly, and then most of the trains from there to Birmingham are run by XC as well!

If this had happened to LNER, say, the effect would be far worse.

I think LNER, much like XC, has suboptimal but perfectly usable alternatives for much of the route, especially if the open access operators can be persuaded to accept tickets.

 

[RyanOPlasty]

Relatively few journeys cannot be made avoiding XC (though doing so would usually add a change of trains). If this had happened to LNER, say, the effect would be far worse.

Living in Nuneaton, most of my rail trips require the use of XC trains even though I do my best to avoid them. I had to drive to Ambergate yesterday.

 

[Snow1964]

Amusing to read the paranoia around whether the Iranians or Russians did it to be honest. Why would they pick *CrossCountry*?

That should be fairly obvious, if pick a large operator (something like 700+ carriages) chances are they invest on better back up and security than a smaller organisation, so going to be much more difficult to attack. So not too big.

Then if going to attack, presumably would time it for when less likely to be security and IT restoration staff available, middle of weekend or overnight would be likely

The other reason is geographic, if you were a rogue in the business of causing disruption, then targeting something nationwide is going to have more effect than something that only operates in a local area.

Trucking company KNP (also known as Knights of Old) is classic example of what can go wrong, cyber attack in 2025 permanently closed it, even though it was big with about 700 employees

 

[Horizon22]

Worth pointing out that in Birmingham there is already a Network Rail Control Centre where XC would not need to do more than agree for a couple of desks to be set aside for them. Given how critical a hub Birmingham New Street, using that control centre on a regular basis would also make a lot of sense. It'd also mean that big things like access to systems and silly things like staff access cards would be already set up for emergency relocation.

Even more handily, Birmingham ROC is a half-hour walk, 10 minute taxi, or 10 minute bus ride from XC's offices.

I’m aware there’s been a lot of of faff, negotiations and false starts about getting TOCs into the ROCs around the West Mids for various reasons. So you’ve still got some sub-optimal practices whereas in most places NR and TOCs are now co-located, even if in different rooms / floors.