All cross country services suspended 23/08/26

[Ze Random One]

Working in IT, we (outside) do not know what has gone wrong. If, for example, their systems are subject to a ransomware attack, then BC/DR facilities are also potentially infected, as can be backups if the ransomware laid low for a while. If there was such a problem the last thing you do is give airtime to your attackers by publicly acknowledging the attack.

If, as is more likely, a key piece of kit has been damaged by the power cut, it may well have been a blind spot in the BC/DR arrangements.

It's very possible that BC/DR has not been tested with everyone actually hoofing it over to the DR ops centre during real operation hours, as enacting a planned full DR scenario has inherent risks of itself. It's more likely that DR was tested during quiet times (Christmas day perhaps, or in the early hours) when of course there will be few/no trains moving and far fewer staff needing access, so perhaps they have a situation where their DR systems are unreliable at full business load.

It's also possible that they've done many DR scenario tests, but the combination of failures they had yesterday was not envisioned.

As another example, I've encountered situations where we've purchased diverse routed telecoms/fibre from two separate companies, who failed to disclose that they were both connected to the same trunk, so it transpired the routing was only diverse as far as the telephone exchange, which left a single point of failure in place until we had an outage on both circuits, and we started asking pointed questions.

Finally, it's 10 days after Microsoft's "patch Tuesday" and is therefore a common weekend for production windows servers to get updated and rebooted (after having let your test systems try out the patches for a week or so, to check they don't break anything important). Perhaps they had a full DR in place, but critical servers in their DR datacentre were being patched at the time of the power cut, and are refusing to come back up until they are recovered from backups. Microsoft have, unfortunately, released Windows Updates over the last few months that have caused instability for some high availability database systems using older versions of Windows Server, and we've been forced to hold back some updates because of it - maybe something like this is compounding the issue.

 

[tumbles]

You would think / hope that contingency plans would include "what would happen if we lost power to HQ / servers". But then this is the railways we are talking about here, contingency planning isn't one of it's strong points.

As someone who works in info security/IT it's easy to carry out continuity testing and plan for incidents but when it happens it can be an entirely different situation to that envisaged. In this instance it could well be third party supplier/vendor issue and guarantees by the vendor have not turned out to be true. Had this years ago where I worked for a company that stored servers in a data centre that we were told that dual redundancy on everything including outbreak/internet. As it turned out we found that they only had one singular internet connection a few years into our contract there when Thames Water cut through the main line some 100 miles away leaving us with a 72 hour blackout. Suffice to say the data centre was empty 3 weeks later and closed completely within 6 months! Backup power testing is certainly one that doesn't get seemingly done enough as we saw elsewhere recently.

 

[dvboy]

As another example, I've encountered situations where we've purchased diverse routed telecoms/fibre from two separate companies, who failed to disclose that they were both connected to the same trunk, so it transpired the routing was only diverse as far as the telephone exchange, which left a single point of failure in place until we had an outage on both circuits, and we started asking pointed questions.

It’s well documented that the BBC once had a failure that took out all of their FM radio networks and it turned out their backup feeds were fed through the same room as the main feeds, and that room overheated.

 

[Mogster]

This seems like the power failure at Manchester from the other week writ large.

Aye, and there’s also the IT related power station problem recently.

Without going 100% tinfoil hat, is there some outside interference linking these issues?

 

[The Chimaera]

Wow, that’s bonkers. surely that can be simplified. Or are unions involved?

The staff and unions don’t like it, it’s purely a management idea to increase productivity. It has actually been recognised as causing problems and said management has coined the trendy phrase of ‘over optimisation of diagrams’ for it. Nothing changes though, who wants to be the manager who makes diagrams less efficient?

 

[Westerndiesels]

I went from Taunton to York in August. Had seats booked but the train layout was not the same and was not sat with wife. Train rammed so much people sat on floor. I could not get to toilet until past Sheffield. Lucky i had water as no trolley, despite them mentioning you could buy from trolley.

Way back we had first class tickets booked but train broke down by Leeds. Got seats on next train luckily but bun fight every stop as people claimed had reserved tickets. Then a fight on board held train up at Bristol Parkway, all the while people full in train including first class aisle. Had to get off at Bristol as cancelled and after getting all sorts of various information another train got us to Taunton via Westbury

Now the poor staff had had a long hot miserable day, fights on at least two trains, breakdowns etc, but my experience with cross country is that the managers or the government, whoever, cannot seem to run a proper, safe, comfortable service.

I live travelling by train but we will never use cross countryvagain until it is sorted. I still have a serious heart condition, operation in October, but the stress, aggro of not being able to easily get to toilet, get refreshments, mean we will clog up the motorways in future.

Such a shame, and I felt sorry for the cross country crew who I felt were let down by others. Had to deal with punch ups, fed up customers etc through no fault of theirs.

Why does this major service get ignored?

I keep seeing pics of Peaks with 10 coaches on old 80s pics, I honestly wish it was like that now.

 

[Adam Williams]

I had to make use of this ticket acceptance today. Decided to drive over and start the journey at Coventry instead of Leamington.

• Coventry gateline: seemed to have no idea what was going on/why I had mentioned I was supposed to be on a CrossCountry service towards Leeds and now wasn't. Told me to scan my (Leamington origin) ticket, and the gateline opened fine (??! I am unconvinced this is because of break of journey). Suggested I take a train back to Leamington.
• Northbound Avanti, catering manager: first tried to tell me the ticket wasn't a 1st class ticket because CrossCountry had set the display value to "Advance Single 1st" instead of "Advance Single (1st Class)" for one particular tier of Advance. Then wanted to check with the train manager as to whether we could get food/drink.
• Northbound Avanti, train manager: tried to claim CrossCountry didn't have first class, then tried to claim they had no catering in first class. I had to explain they would serve you a thai curry and drinks before he relented.
• LNER service from Kings Cross: no issues whatsoever.

Booked reservations on both the Avanti Pendo and the Azuma. The reservation display in the LNER train took 1h20m to reflect the reservation I had booked, and just persistently showed Available until Newark Northgate! Very poor tech.

 

[johntea]

As someone that also (used) to work in IT for the NHS there is also potentially just one IT person 'on call' over the weekend who will obviously do their best to get things back up and running but you can't expect them to resolve everything without the wider team being available to help out

I've also come across *many* systems over the years promising 'high availability' that were about as high availability as a chocolate fireguard when it came to it, in fact sometimes made things worse trying to troubleshoot as the system was fighting with itself as to which node was live and which should be in standby! (in technical terms a 'split brain')

 

[43055]

I've just been over to Derby station, and all XC services are showing as cancelled, including the Cardiff-Nottingham, hence no way of getting to Birmingham this afternoon, unless you want a scenic trip via Stoke.

Word has obviously got out, as the place is practically deserted.

Burton was more dead this afternoon when I went by on the bus so maybe news has spread.

Had a few try and get my bus from it all. One guy was trying to get to Tamworth with it not the easiest on a Sunday by bus from Derby.

 

[railpigeon]

I am due to travel Leeds to Newton Abbot tomorrow. I have just messaged the Twitter teams at Cross Country, GWR, and LNER about acceptance tomorrow. They have replied saying 'check tomorrow'. But fair play to them, all three Twitter teams replied within a couple of minutes.

 

[rmHawk765]

That is pathetic, but nothing about the abysmal information system on our railways today surprises me anymore.

It amazes me that we are fast approaching 2030, yet we still cannot provide integrated, detailed information to passengers in disruption. No, we cannot just become Switzerland and never have issues, which is common sense to anyone who even slightly understands the complexity of our network. The average passenger doesn't because with a complex network you'd expect there to be robust systems in place when something breaks - but alas, there are not.

How difficult can it be to provide detailed audio and graphical information at major stations so passengers are not just told to go away? Nevermind whether an entire operator should go down because of a power outage in a single office tower, there should be disruption information systems that can scale to even that bizzare of a failure. It's not even like nationalisation is necessary to support just communicating with passengers.

 

[tumbles]

Working in IT, we (outside) do not know what has gone wrong. If, for example, their systems are subject to a ransomware attack, then BC/DR facilities are also potentially infected, as can be backups if the ransomware laid low for a while. If there was such a problem the last thing you do is give airtime to your attackers by publicly acknowledging the attack.

If, as is more likely, a key piece of kit has been damaged by the power cut, it may well have been a blind spot in the BC/DR arrangements.

It's very possible that BC/DR has not been tested with everyone actually hoofing it over to the DR ops centre during real operation hours, as enacting a planned full DR scenario has inherent risks of itself. It's more likely that DR was tested during quiet times (Christmas day perhaps, or in the early hours) when of course there will be few/no trains moving and far fewer staff needing access, so perhaps they have a situation where their DR systems are unreliable at full business load.

It's also possible that they've done many DR scenario tests, but the combination of failures they had yesterday was not envisioned.

As another example, I've encountered situations where we've purchased diverse routed telecoms/fibre from two separate companies, who failed to disclose that they were both connected to the same trunk, so it transpired the routing was only diverse as far as the telephone exchange, which left a single point of failure in place until we had an outage on both circuits, and we started asking pointed questions.

Finally, it's 10 days after Microsoft's "patch Tuesday" and is therefore a common weekend for production windows servers to get updated and rebooted (after having let your test systems try out the patches for a week or so, to check they don't break anything important). Perhaps they had a full DR in place, but critical servers in their DR datacentre were being patched at the time of the power cut, and are refusing to come back up until they are recovered from backups. Microsoft have, unfortunately, released Windows Updates over the last few months that have caused instability for some high availability database systems using older versions of Windows Server, and we've been forced to hold back some updates because of it - maybe something like this is compounding the issue.

Funnily enough we’ve experienced the situation with two separate fibre lines being routed through same data centre against the knowledge of our supplier who in this instance had triple redundancy and found out the hard way that 2/3 could go down with a single failure.

There is a known issue with the cumulative update for servers this month causing them to freeze so it’s entirely possible

 

[Shrewbly]

It looks like the Birmingham - Leicester service may be about to restart. A couple of ECS moves in progress from Tyseley and Barton under Needswood.

 

[ox_chap]

Presumably there is nothing in the failure that would stop a train physically moving? So if still a problem tomorrow, something like crews of two could run a Brum-Reading shuttle? Similarly elsewhere. So nothing fancy, just shuttles where XC provide key links. Even if it involves a paper-based system?

 

[43055]

It looks like the Birmingham - Leicester service may be about to restart. A couple of ECS moves in progress from Tyseley and Barton under Needswood.

The one from Barton looks like it's for a Birmingham - Nottingham service at 1749.

 

[TUC]

They're running perfectly fine

So what conclusions should we draw from this in terms of GC vs Cross Country's ability to operate?

 

[Ze Random One]

Aye, and there’s also the IT related power station problem recently.

Without going 100% tinfoil hat, is there some outside interference linking these issues?

I think it's more likely to be sustained heat stress-testing our infrastructure. Remember that air conditioning and the like will have been running harder, for longer, across the country, plus the substations may not have been designed for wave after wave of days over 35°C. Add to this the soil moisture deficit may be causing the same problems for underground electric cables as being experienced by SE Water and the various derailed trains. Furthermore, the heat may be penetrating further into the soil, allowing underground cables to heat further than normal, causing them to warp.

 

[dvboy]

So what conclusions should we draw from this in terms of GC vs Cross Country's ability to operate?

Similar reason to why Stansted - Peterborough can run. It’s not connected to the wider network through Birmingham and is probably small enough that it can be managed by one person using manual processes remotely.

 

[Silver Cobra]

I’m at Oxford station. XC are not shown on the screens at all (no word “cancelled”.). Lots of very confused people. Surely as a minimum the train should be shown as not running. Once folk talk to the station then all is explained - and the staff here are fantastic as always. Nevertheless…

This is something Thameslink often do when there is major disruption to their services. They remove the cancelled services from the departure boards, to only show services that are confirmed as running. Granted they do also explain this with an additional message on the departure boards.

 

[Class 317]

looking at real time trains a Birmingham to Nottingham cross country looks to be running at the moment.

Hopefully a sign of recovery.

 

[sikejsudjek]

According to a reply by Xcountry on Facebook just before today's shambles, they are experiencing significant delays in processing delay repay claims. So even getting a refund could take weeks. At what point does the government take the contract back? Trains are still regularly cancelled because of a lack of drivers.

 

[45Fox]

XC services between Birmingham & Leicester are definately starting to run.

Confirmed departures are 18.20 (1P26) & 20.20 (1P30) from Leicester and 17.52 (1K25) & 21.52 (1M85) from Birmingham, others may be running but no details as yet.

It would also appear that a service start-up between Birmingham and Derby/Nottingham is imminent but no firm details as yet.

*edit* 1M73 for Nottingham departed Birmingham right time at 17.49.

 

[py_megapixel]

tried to tell me the ticket wasn't a 1st class ticket because CrossCountry had set the display value to "Advance Single 1st" instead of "Advance Single (1st Class)" for one particular tier of Advance.

What?!? I'm surprised anyone would notice that, let alone think it was significant. There are variances between different apps' display of e-tickets that are far worse than that anyway...

 

[ox_chap]

Just got off a Cotswolds line train at Oxford. Much more full than usual before even getting to Oxford, so maybe XC passengers are trying via Worcester. Arrived to five deep rows of people waiting at Oxford…. What must international visitors to the UK think?

 

[liamf656]

XC services between Birmingham & Leicester are definately starting to run.

Confirmed departures are 18.20 (1P26) & 20.20 (1P30) from Leicester and 17.52 (1K25) & 21.52 (1M85) from Birmingham, others may be running but no details as yet.

It would also appear that a service start-up between Birmingham and Derby/Nottingham is imminent but no firm details as yet.

Just between Derby and Birmingham it seems to be

 

[1D54]

Looks at though 18:16 NOT-BHM is about to leave with 220001 and 221119 doing the honours.

 

[liamf656]

Looks at though 18:16 NOT-BHM is about to leave with 220001 and 221119 doing the honours.

Service starting from Derby vice Nottingham

 

[GordonT]

At what point does the government take the contract back? Trains are still regularly cancelled because of a lack of drivers.

Would stripping the operator of the contract make much difference to reliability of the operation, or alter the day-to-day providers of the services, or resolve any insufficency of drivers? Perhaps in the Cross Country operation we have a poisoned chalice with no-one particularly lusting to take it over?

 

[AdamWW]

What must international visitors to the UK think?

That things sometimes go wrong as happens in other countries?

 

[Dave61]

For everyone wondering about Disaster Recovery/Business Continunity, this sort of stuff in IT can cost some serious money not only to impliment but also the need for regular testing along with upgrades for time to time - how much extra are you willing to pay for your ticket to cover the cost?

And each senario will have been considered on a likely hood of happening v cost of implimenting a DR/BC for it v cost of not having it.

In this case (from reading the thread) it appears to have been a power failure in a building shared with other businesses. Sharing a building will restrict your options and you will mostly likely be faced with a dead building (power wise) with no lighting, air conditioning, working lifts, facilities etc. To maintain normal operation you will need *another* building somewhere that is linked to your IT setup in the first building - nothing else would work in this senario.

So you have a duplicate facility, maybe sitting there gathering dust until needed (yes you can rent places but there is still an ongoing cost and it takes longer to get going), or you can use both. But now you have 2 buildings to pay for, 2 sets of IT kit to upgrade/patch (you must keep them in sync) not to mention regular DR testing.

Against this how many times as XC suffered from this outage? My reading of the thread suggests it is the first time, so that is X years of duplicated kit. So I would have expected the Management assesment being something along the lines of "it will cost far more than we would loose if it happened, and the chances are minimal" and here we are today.
On DR/BC part has worked however - the twitter feeds are working just fine, so they can tick that box (until it goes TITSUP of course).