• Our new ticketing site is now live! Using either this or the original site (both powered by TrainSplit) helps support the running of the forum with every ticket purchase! Find out more and ask any questions/give us feedback in this thread!

Worldwide IT Outage (Crowdstrike update)

Status
Not open for further replies.

Howardh

Established Member
Joined
17 May 2011
Messages
9,774
One of the newly reported remedies by Microsoft is now to reboot machines 15 to 20 times

Is there a shortcut to "hit with hammer"?

Basically, I wouldn't know where to start and there aren't any 7 - 9 yr olds around here to help!! Noted that it's only "virtual machines" that MS refer to so how many businesses run virtual? Lots? Few??
 
Sponsor Post - registered members do not see these adverts; click here to register, or click here to log in
R

RailUK Forums

CaptainHaddock

Established Member
Joined
10 Feb 2011
Messages
2,716
Cash is of no use when all the tills go down
An "Arkwright"-style till would be unaffected by an IT problem, wouldn't it? Alternatively a small business such as a cafe or micropub could, in case of emergencies, just have a cash "jar", with enough change to enable customers to pay by cash.
 

Spaceship323

Member
Joined
24 Jan 2020
Messages
617
Location
Nuneaton Trent Valley
There was massive problems on the ECML out of KX last night, all trains north cancelled at one point, wonder if that had anything to do with it? "Signalling at Welyn" was said to be the cause. Took me 4 hours to get home via Liverpool Street
Signal failure at Welwyn, trains going via Hertford loop as usual!
 

Howardh

Established Member
Joined
17 May 2011
Messages
9,774
An "Arkwright"-style till would be unaffected by an IT problem, wouldn't it? Alternatively a small business such as a cafe or micropub could, in case of emergencies, just have a cash "jar", with enough change to enable customers to pay by cash.
One of the difficulties for small businesses is that if they take cash, they could be a very long way from a bank or post office to cash it in, which literally costs time and money. When I worked in industry the lads delivering would take cash, bring it back to be counted and accounted for, and was my job to go to the bank before closing to cash it all in. Long gone these days!
 

Taunton

Veteran Member
Joined
1 Aug 2013
Messages
12,255
In my book, they belong to the “just in time” brigade. Always carry enough cash to get you home at the very least…
That's a variant on a very old expression, that on a date a girl should always have enough cash to suddenly get themselves a taxi to get home - if needs be!

I did notice that in the office this morning in central London there were genuine concerns from those (several) who had parked at their local station and the parking machine rejected their credit card, that the railway would fine them an inordinate amount for not paying for parking. How on earth has the railway even come to this customer-unfriendly perception.
 

OscarH

Established Member
Joined
15 Sep 2020
Messages
1,274
Location
Crawley
I think I’d rather take the chance on an occasional wonky update than continue to run software that has vulnerabilities that will never be fixed.
This isn't even a wonky update of the OS anyway, there's a good chance that if we were in Windows 7 days and the same piece of 3rd party software did that same update it would fail in the same way
 

nlogax

Established Member
Joined
29 May 2011
Messages
6,016
Location
Mostly Glasgow-ish. Mostly.
Noted that it's only "virtual machines" that MS refer to so how many businesses run virtual? Lots? Few??

As someone working for a very large hypervisor vendor, I can't remember the last time I saw Windows Server running on bare metal in a customer site. VMs all the way for larger customers.
 

bleeder4

Member
Joined
19 Jan 2019
Messages
792
Location
Worcester
Which is why it's sensible to keep a cask "float" of around £100 at home so you're covered in the event of incidents such as today's.
Indeed. There are 3 supermarkets near me and every single one has been cash only today due to this. Fortunately, I am in the habit of keeping a small wad of notes in my safe, so was able to do my shopping as planned. The cashier who served me said she had been told by her manager it could take several days to come back, as head office only had a few IT staff left and they were all having to drive to each store in the country to fix the affected machines on-site, as it can't be done remotely.
 

AM9

Veteran Member
Joined
13 May 2014
Messages
16,029
Location
St Albans

Which is why it's sensible to keep a cask "float" of around £100 at home so you're covered in the event of incidents such as today's.
Yes that is my answer. All these stupid commnets about cash being useless without computer driven ATMs, we managed without them for 200 years by using banks that had humans sitting at counters. Of course those have been cut back drastically, but temporarily it would be possible to get by until the digital heaven that some worship got its act together. Meanwhile, for those not so obsessed with banning cash from their live, it was simple to just go to a drawer at home and pocket £40 each and visit the stricken Waitrose to get our essentials.
This event should silence the 'cash is obsolete' and 'why should cash be maintained when I don't need it' mob for a while.
 

Ediswan

Established Member
Joined
15 Nov 2012
Messages
3,421
Location
Stevenage
The only thing not working in the local Co-op was Member Prices. Staff were applying the discounts manually, trusting the customer.
 

jfollows

Established Member
Joined
26 Feb 2011
Messages
10,096
Location
Wilmslow
My first paid employment in IT was in 1980, and I worked in IT from 1984 to 2015. It's a sobering thought that, when I started, we'd say things like "at least nobody died" because of IT failings. Sobering because that won't be true because of today's problems, which were potentially caused by one person making a single mistake which shouldn't have been made.
 

bleeder4

Member
Joined
19 Jan 2019
Messages
792
Location
Worcester
My first paid employment in IT was in 1980, and I worked in IT from 1984 to 2015. It's a sobering thought that, when I started, we'd say things like "at least nobody died" because of IT failings. Sobering because that won't be true because of today's problems, which were potentially caused by one person making a single mistake which shouldn't have been made.
I don't have your expertise, but surely if IT infrastructure is set up in such a way that it is possible for a single person's mistake to have such an impact, then that infrastructure is set up wrongly? For example, was there not a review process at Crowdstrike, so that at least one other person had to peer review their colleague's code before it was released to update servers? Was there not someone (or several people) at Microsoft who manually checked the update first before allowing it to be released to Windows users? I would have thought there must be checks and reviews, precisely to stop one person's mistake having such an effect.
 

Magdalia

Established Member
Joined
1 Jan 2022
Messages
7,378
Location
The Fens
Yes that is my answer. All these stupid commnets about cash being useless without computer driven ATMs, we managed without them for 200 years by using banks that had humans sitting at counters. Of course those have been cut back drastically, but temporarily it would be possible to get by until the digital heaven that some worship got its act together. Meanwhile, for those not so obsessed with banning cash from their live, it was simple to just go to a drawer at home and pocket £40 each and visit the stricken Waitrose to get our essentials.
This event should silence the 'cash is obsolete' and 'why should cash be maintained when I don't need it' mob for a while.
Absolutely.

This morning I was able to pay by card at my local Waitrose, but I had enough cash with me just in case. That cash comes periodically from counter withdrawals at my local Building Society branch. It is resilient too, my passbook does include withdrawals recorded in manuscript.
 

dk1

Veteran Member
Joined
2 Oct 2009
Messages
19,848
Location
East Anglia
Took a shed load of cash out eith me today rather than the usual emergency tenner. No issues whatsoever with any payment or apps.
 

jfollows

Established Member
Joined
26 Feb 2011
Messages
10,096
Location
Wilmslow
I don't have your expertise, but surely if IT infrastructure is set up in such a way that it is possible for a single person's mistake to have such an impact, then that infrastructure is set up wrongly? For example, was there not a review process at Crowdstrike, so that at least one other person had to peer review their colleague's code before it was released to update servers? Was there not someone (or several people) at Microsoft who manually checked the update first before allowing it to be released to Windows users? I would have thought there must be checks and reviews, precisely to stop one person's mistake having such an effect.
It wasn’t Microsoft who released this, but otherwise I agree with you. Or rather, they weren’t involved in any way.
 

GRALISTAIR

Established Member
Joined
11 Apr 2012
Messages
10,492
Location
Preston Lancs
Yes that is my answer. All these stupid commnets about cash being useless without computer driven ATMs, we managed without them for 200 years by using banks that had humans sitting at counters. Of course those have been cut back drastically, but temporarily it would be possible to get by until the digital heaven that some worship got its act together. Meanwhile, for those not so obsessed with banning cash from their live, it was simple to just go to a drawer at home and pocket £40 each and visit the stricken Waitrose to get our essentials.
This event should silence the 'cash is obsolete' and 'why should cash be maintained when I don't need it' mob for a while.
I am obsessive about cash and I live in the USA but always and currently have with my passport and wallet about 200$ US, 50£ , 50 € and 20$ Canadian. It is a useful emergency imho.
 

bleeder4

Member
Joined
19 Jan 2019
Messages
792
Location
Worcester
It wasn’t Microsoft who released this, but otherwise I agree with you. Or rather, they weren’t involved in any way.
The Crowdstrike update was delivered to machines via Microsoft's Windows Update service though. It didn't get delivered via an update within the Cloudstrike software. My point was that, rather than Microsoft blindly accepting an update from a third party into their Windows Update service, did they not manually check/test it themselves first?
 

jfollows

Established Member
Joined
26 Feb 2011
Messages
10,096
Location
Wilmslow
The Crowdstrike update was delivered to machines via Microsoft's Windows Update service though. It didn't get delivered via an update within the Cloudstrike software. My point was that, rather than Microsoft blindly accepting an update from a third party into their Windows Update service, did they not manually check/test it themselves first?
Agreed - and probably not would be my guess. A bit like the FAA allowing Boeing to certify itself, perhaps?
 

Starmill

Veteran Member
Joined
18 May 2012
Messages
27,228
Location
Bolton
An "Arkwright"-style till would be unaffected by an IT problem, wouldn't it? Alternatively a small business such as a cafe or micropub could, in case of emergencies, just have a cash "jar", with enough change to enable customers to pay by cash.
Sometimes small businesses would keep a cash box, a book for receipts and a calculator somewhere for that.

However there's legwork and hassle involved with that, and too many modern British consumers just won't buy if a business is cash only, so it's necessary to do both.
 

JamesT

Established Member
Joined
25 Feb 2015
Messages
4,850
The Crowdstrike update was delivered to machines via Microsoft's Windows Update service though. It didn't get delivered via an update within the Cloudstrike software. My point was that, rather than Microsoft blindly accepting an update from a third party into their Windows Update service, did they not manually check/test it themselves first?
Do you have a reference for that?
The statement at https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/ includes:
CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
Which to me implies they’re pushing the updates directly and MS Update isn’t involved at all.

Windows Update generally isn’t involved in pushing updates for third-party software. The nearest they get is shipping drivers provided by the hardware manufacturer. Those do get tested first, which is why they’re generally behind what you get directly.
 

jfollows

Established Member
Joined
26 Feb 2011
Messages
10,096
Location
Wilmslow
The other thing which has come to the fore is the prevalence of excuses, maybe made by less reputable companies, Ryanair for example, making a point about "because of a failure of third party software, we're unable to ....."

No, you chose to use Windows, in whatever way you did, and if it doesn't work it's your problem and your responsibility. Sort out your issues with third parties in the background, we don't need to know.
 

pokemonsuper9

Established Member
Joined
20 Dec 2022
Messages
4,601
Location
Greater Manchester
The other thing which has come to the fore is the prevalence of excuses, maybe made by less reputable companies, Ryanair for example, making a point about "because of a failure of third party software, we're unable to ....."
I don't think there's anything wrong with the statement, it's accurate, a third party sent out a broken update and broke everything, are you saying it's their fault for using a trusted software provider used in almost every industry?
 

Ediswan

Established Member
Joined
15 Nov 2012
Messages
3,421
Location
Stevenage
Windows Update generally isn’t involved in pushing updates for third-party software. The nearest they get is shipping drivers provided by the hardware manufacturer. Those do get tested first, which is why they’re generally behind what you get directly.
Agreed. It could be that CrowdStrike use Microsoft Installer packeges, but anybody can use those.
 

jfollows

Established Member
Joined
26 Feb 2011
Messages
10,096
Location
Wilmslow
I don't think there's anything wrong with the statement, it's accurate, a third party sent out a broken update and broke everything, are you saying it's their fault for using a trusted software provider used in almost every industry?
To me it can come across as evasive, but of course there's a good argument for being truthful. I guess for me it's about how it's phrased, whether it comes across as trying to shift the blame rather than taking responsibility.
 

GatwickDepress

Established Member
Joined
14 Jan 2013
Messages
2,588
Location
Leeds
No issues with the payment systems at work today - a certain supermarket that utilises a loyalty scheme named for sugary secretions of the floral kind - or any other stores in the region, barring activating or using physical gift cards.

It seemed to have the most impact on back end systems like inventory, reports, and other such things. We did chortle at the sales numbers claiming we only made £2.20 in sales yesterday though!

I don't think there's anything wrong with the statement, it's accurate, a third party sent out a broken update and broke everything, are you saying it's their fault for using a trusted software provider used in almost every industry?
I agree. I don't see much wrong with the statement. It clarifies that the issue is not Ryanair's own making (the pilot didn't accidentally lose the plane in a strip poker game or something) but an issue with a supplier, something that happens in every single industry.

It's also important to note that people like knowing the reasons behind something that inconveniences them. It's one of those human nature things.
 
Status
Not open for further replies.

Top