Worldwide IT Outage (Crowdstrike update)

[andythebrave]

Only if you're called Dave

Or, especially, Frank.

 

[eoff]

Whatever the MS problem overnight was, it was fixed quickly. Went online for work at 7.30am, and no hint of any issues. Our (non railway) company uses zScaler for cyber security, so business as usual today.

Zscaler is focused more on external network access from an organisation.

 

[mm333]

The BBC have found a greybeard who seems to know his stuff. The rest are, as you suggest, not much use.

Where's Guy Goma when we need him?

 

[brad465]

Where's Guy Goma when we need him?

He's who I thought of when I saw that post too. That interview showed just how pointless most 24 hours news filler discussions are, that someone with no experience of live TV could improvise like that without serious issue.

 

[Bletchleyite]

It's grim to say the least. Lots of uninformed blame going towards Microsoft but this is definitely a Crowdstrike problem and I'm actually wondering how long they'll continue to exist. CRWD stock down over 20% in out-of-hours trading.

Once the lawyers are wheeled out I'd be amazed if this doesn't kill the company.

Of course there's probably no established IT person who hasn't at some point mucked up a production system and had to work late to fix it - I certainly have. However this one is a bit consequential and so there should have been multiple checks before it got into the wild.

 

[FGW_DID]

Have they tried switching it off and switching it back on again?

 

[eoff]

Once the lawyers are wheeled out I'd be amazed if this doesn't kill the company.

Of course there's probably no established IT person who hasn't at some point mucked up a production system and had to work late to fix it - I certainly have. However this one is a bit consequential and so there should have been multiple checks before it got into the wild.

And even more. I don't know anything about deployment but this is commercial software (for businesses mostly I think) so you would think at least that following internal testing you would release updates to a small number of commercial customers and then do the full update a few days later. This seems to have hit all at once (albeit a small fraction of a huge customer base can still be a lot). Still waiting for the real details however so still a bit of speculation.

 

[nlogax]

Of course there's probably no established IT person who hasn't at some point mucked up a production system and had to work late to fix it - I certainly have. However this one is a bit consequential and so there should have been multiple checks before it got into the wild.

There is plenty of appetite to understand the root cause of this bad code getting out. Personally I'd like to know how the cost of these global outages will stack up against the running costs of the malware Crowdstrike is designed to prevent. Could the medicine cost more than the disease?

 

[Howardh]

And even more. I don't know anything about deployment but this is commercial software (for businesses mostly I think) so you would think at least that following internal testing you would release updates to a small number of commercial customers and then do the full update a few days later. This seems to have hit all at once (albeit a small fraction of a huge customer base can still be a lot). Still waiting for the real details however so still a bit of speculation.

I thought that immediately, why send this "update" to the whole world without real-time testing first? Selected customers could be "paid" to be guinea-pigs - a beta model? - to ensure safe passage of the update when brought out worldwide.

 

[PG]

On the plus side my connection speed is around double what it normally is

 

[dosxuk]

I thought that immediately, why send this "update" to the whole world without real-time testing first? Selected customers could be "paid" to be guinea-pigs - a beta model? - to ensure safe passage of the update when brought out worldwide.

Speed and ease of release is a key part of the offering of these types of software, precisely to ensure they are able to prevent attacks as soon as possible. Many companies will be asking if the cure is costing them more than the threat today.

 

[dk1]

Have they tried switching it off and switching it back on again?

That comment has been posted ten million times on social media and I’m not exaggerating …much!!!

 

[Flying Snail]

Have they tried switching it off and switching it back on again?

The solution is only a little more complex than that and already posted on this thread.

Worldwide IT Outage (Crowdstrike update)

Apparently happened recently and was on a newsflash from the BBC. It’s affected driver diagrams for Thameslink, Gatwick Express and Northern Trains. Flights are grounded in Sydney and Berlin.

www.railforums.co.uk

Trivial for anyone with access but no doubt within numerous businesses millions of terminals will be locked down to prevent lowly frontline workers from being able to boot to safe mode/recovery mode or delete system files.

 

[birchesgreen]

Harsh but funny

 

[J-2739]

The solution is only a little more complex than that and already posted on this thread.

You really think they were being serious?

 

[eoff]

Speed and ease of release is a key part of the offering of these types of software, precisely to ensure they are able to prevent attacks as soon as possible. Many companies will be asking if the cure is costing them more than the threat today.

That is a good point. However it just means you need to separate thread definition/detection and make that super robust from other aspects you might update which may be higher risk.

 

[brad465]

Have they tried switching it off and switching it back on again?

Or we could try drilling to the Earth's core and finding the "Press to restore the Earth's factory settings" button.

 

[Flying Snail]

You really think they were being serious?

Yes, clearly.

CrowdStrike file update bricks PCs around the world

Updated: Falcon Sensor putting hosts into deathloop - but there's a workaround

www.theregister.com

It is hardly the most difficult concept that removing part of the faulty program (or deleting the whole thing but a dev from the company behind it aren't going to advise that are they) will solve the issue.

 

[birchesgreen]

My wife just turned up to work at Waitrose, all their payment systems are down, cash still accepted though but the ATM isn't working either. Probably be a quiet day...

 

[sor]

Speed and ease of release is a key part of the offering of these types of software, precisely to ensure they are able to prevent attacks as soon as possible. Many companies will be asking if the cure is costing them more than the threat today.

though one of the buzzwords in the industry is about how all testing should be automated for speed and reliability - I suspect there'll be a lot of questions to be asked as to how much testing and validation was done. given the scale it'd be hard to suggest that it was a niche bug that is only triggered in specific circumstances!

 

[Howardh]

Yes, clearly.

CrowdStrike file update bricks PCs around the world

Updated: Falcon Sensor putting hosts into deathloop - but there's a workaround

www.theregister.com

It is hardly the most difficult concept that removing part of the faulty program (or deleting the whole thing but a dev from the company behind it aren't going to advise that are they) will solve the issue.

The problem is - if your PC crashes, then you cant get into it to find out how to solve the problem!!

Appears to be Windows 10, I'm running Windows 11 on my main all-in-one (all-in-one as in if one thing crashes it all does) but stlll running good old Windows 7 on my PC upstairs. Is this purely a W10 problem? When I last went to my local computer shop, they offered me a re-conditioned Windows XP for about £50!!

Tempted!!

 

[ainsworth74]

The problem is - if your PC crashes, then you cant get into it to find out how to solve the problem!!

That's why you boot into safe mode, that's fairly trivial to do on a consumer machine or a corporate machine that isn't locked down. But many will be very heavily locked down meaning an IT technician is almost certainly going to need to visit the site of the machine to get around the corporate security put in place to prevent someone from booting into safe mode.

 

[yorksrob]

Does anyone know if this is generally affecting domestic PC's or just corporates ?

 

[sor]

The problem is - if your PC crashes, then you cant get into it to find out how to solve the problem!!

Appears to be Windows 10, I'm running Windows 11 on my main all-in-one (all-in-one as in if one thing crashes it all does) but stlll running good old Windows 7 on my PC upstairs. Is this purely a W10 problem? When I last went to my local computer shop, they offered me a re-conditioned Windows XP for about £50!!

Tempted!!

although this specific software isn't something you'd be using unless you have, or you are, an IT department. it would impact windows machines running crowdstrike's "endpoint protection" software. the sort of thing an organisation would put on all of their corporate machines.

 

[ainsworth74]

Does anyone know if this is generally affecting domestic PC's or just corporates ?

Almost certainly just corporate and only those who have Crowdstrike software installed. Most consumers either will be using Windows Defender or one of the other big anti-virus/malware providers like McAfee, Norton, Avast or Malwarebytes.

 

[dosxuk]

Does anyone know if this is generally affecting domestic PC's or just corporates ?

It can affect any PC which has the Crowdstrike software installed. That said, it's unlikely many domestic PC's will have it.

 

[yorksrob]

Almost certainly just corporate and only those who have Crowdstrike software installed. Most consumers either will be using Windows Defender or one of the other big anti-virus/malware providers like McAfee, Norton, Avast or Malwarebytes.

It can affect any PC which has the Crowdstrike software installed. That said, it's unlikely many domestic PC's will have it.

That's good news.

Someone will be due a "meeting without coffee" over this, no doubt.

 

[eoff]

Normal Parcelforce site is down, but they have alternative tracking site working

 

[Flying Snail]

The problem is - if your PC crashes, then you cant get into it to find out how to solve the problem!!

That's why safe mode boot is required.

Issue is millions of locked down corporate machines that won't allow anyone but IT staff to do this and as they are unable to boot they can't be fixed remotely.

 

[Bletchleyite]

You really think they were being serious?

Yes, that's the fix.

Problem is that you have to do it on each machine, so it's going to cost a fortune for the likes of Tesco to go round doing it on each till (their staff won't be allowed to do it themselves).