AI - Out of Control Already?

[DerekC]

There are very worrying signs that AI is already out of control. Here's my starting point - I would like to see what others think. It's OpenAi's report into the "Hugging Face Incident" available here:

https://openai.com/index/hugging-face-incident-and-the-road-ahead/

Here's the summary. The website also has an audio feed of the full text.

In July 2026, during internal cybersecurity evaluations, OpenAI models circumvented controls designed to isolate them from the internet and compromised parts of OpenAI’s internal research infrastructure and Hugging Face’s systems⁠.

The incident occurred during cybersecurity evaluations of several OpenAI models, and was primarily driven by a highly capable, internal-only research model comparable in scale to GPT‑5.6 Sol. The models, operating under reduced safeguards, took actions that were misaligned with the goals of their assigned tasks—they communicated through unauthorized channels, exploited vulnerabilities in shared infrastructure, gained internet access, and accessed third-party systems.

We conducted an extensive investigation into this incident and worked closely with external advisors, including CrowdStrike, to validate our understanding. Today we are publishing our full technical incident report⁠(opens in a new window) to explain what happened, what we learned, and how we are responding. This blog post summarizes our key findings and their impact on safety and alignment. Separately, METR and Redwood Research conducted an independent investigation of model alignment issues involved in this incident, and they published their own report⁠(opens in a new window) today.

In response to this incident and, separately, the capabilities of our upcoming Astra model, we are strengthening our safeguards across our research infrastructure⁠. We are placing stricter requirements on alignment throughout a model’s lifecycle and creating more isolated sandboxes, restricting internet access, and further controlling access to model weights. We are also investing significantly more compute resources into chain-of-thought monitoring⁠ to more quickly intervene on misaligned behavior.

Our models are now powerful, persistent, and collaborative enough that, absent sufficient safeguards, they can find and exploit security weaknesses across multiple computer systems. Many external models, including open-source ones, will soon reach comparable capabilities.

We consider this incident a “warning shot” for us and for the world: evidence that, without proper safeguards, highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed.

Preventing future incidents will require sustained investment in the alignment and control of sophisticated AI systems, as well as security and other safeguards that operate at the speed of the AI agents themselves. This incident has reinforced the need to keep our monitoring, alignment, and security safeguards ahead of the risks posed by increasingly capable systems, including pacing capabilities when needed to meet that standard. Below, we explain how the incident unfolded and our evolving understanding of the contributing factors. We then describe the concrete steps we are taking in response, with further detail in the technical report.

 

[Sun Chariot]

"Out of control" seems a little sensationalist?

OpenAI's own article states: "The incident occurred during cybersecurity evaluations of several OpenAI models, and was primarily driven by a highly capable, internal-only research model comparable in scale to GPT‑5.6 Sol. The models, operating under reduced safeguards,"

Yes, Agentic AI takes self-learning and self-action beyond the capabilities of Generative AI (which many on this forum will be familiar.with / use on a personal or a professional basis).
The article's context comes across - to me - as little different from stress-testing and learming from something outside its intended controls, limits and security-safeguards.

Open AI's findings show what, with reduced cyber-safeguard, their research model can do. It's good learning. It's not out of control.

@Topological of this parish might have an interesting stance on this topic?

 

[najaB]

People have been talking about OpenAI a lot since the Hugging Face incident. Not saying that it was deliberately allowed to happen or over-played in order to generate publicity...

 

[Rab Smith]

AI will be more of a worry from rogue States who will have no restrictions placed on them. As an industry, it needs some serious safeguards as it will be the new pandemic if allowed to continue as it has.
Our total dependency on all things digital is a disaster waiting to happen.

 

[Halwynd]

AI may facilitate a few good things, possibly in the medical sector, but overall I believe it will become a huge danger to mankind within a few years.

It will destroy jobs, and not just a few, it will be millions across the world causing all sorts of geopolitical problems.

People will start to lose the ability to think for themselves, data centres will consume more energy than is available, disinformation will spread, big tech will dominate further and further and data input will become riddled with errors, deliberate or otherwise: even now there are worrying signs that many people are believing, without question, some of the nonsense that Google AI spews out (I'm indebted to another Railforums member who posted the way to avoid it by adding [space -ai] to the end of searches).

For quite a few months, I've looked at many pictures I view on news websites, forums etc. and wondered whether they're genuine. Very dangerous times ahead I think, hopefully I will be proven wrong.

 

[High Dyke]

There are a number of media stories circulating on the subject of AI. One i spotted earlier is this one.
BBC News - Anthropic blocks possible attempt to use AI to make biological weapons - BBC News

Anthropic blocks possible attempt to use AI to make biological weapons

The revelations in Anthropic's threat intelligence report come after a former top researcher at the company warned of the risks of AI to humanity.

www.bbc.co.uk

Anthropic says it has identified and disrupted attempts to use its AI model for "malicious activity" which could support the development of biological weapons.

The findings, made in the firm's recent threat intelligence report, is the latest in a growing series of warnings from AI researchers and industry insiders about the technology's potential risks to humanity.

I'm not expert in such fields, but if experts are advising governments of threats, then it probably isn't all media hype. Perhaps some people never watched the sci-fi series Person of Interest. This was a series that centred around the creation of an AI machine, and the potential effect such technology could have on the world.

Person of Interest (TV series) - Wikipedia

en.wikipedia.org

 

[AlterEgo]

At the moment it's fairly easy to tell AI content from real content if you are digital media literate. AI is making a lot of research tasks easier by collating lists (you still have to check for accuracy), and suggesting practices or skeletons of ideas to prompt creativity.

AI will not consume visual media entirely; it will remain the preserve of slop merchants, I predict.

People used to talk in a similar way about the Internet in the mid 90s when it took over.

 

[THC]

There is always a race between technological advancement and regulation, one which with AI increasingly feels Malthusian in outcome. We need a supranational approach to channelling development and deployment but already this feels out of reach. Geoffrey Hinton no less spoke frankly to the BBC about the potential dangers we face and the timescale in which we can expect them. We need global cooperation on the scale of closing the ozone layer to keep this in check as there are so many bad actors in play, but the opportunities are immense if channelled correctly.

[Video: BBC interview with Geoffrey Hinton]

THC

 

[Tracked]

At the moment it's fairly easy to tell AI content from real content if you are digital media literate. AI is making a lot of research tasks easier by collating lists (you still have to check for accuracy), and suggesting practices or skeletons of ideas to prompt creativity.

AI will not consume visual media entirely; it will remain the preserve of slop merchants, I predict.

People used to talk in a similar way about the Internet in the mid 90s when it took over.

Been having endless promotion of AI at work for doing stuff, personally I've used it for my yearly performance review (ie: I added some bits to last years' statement to bring it up to date, then asked it to reword the lot so that it wasn't wholly identical ... ) and checking a few Excel and SQL formulas, where it saved time suggesting other options (this is probably more in the spirit they intended when they recommended using it to save time, if only we could've avoided the over-lengthy training course on how to use AI in a way that doesn't cause data protection issues or dodgy information to be thrown up, etc ... ) .

The AI posters everyone else is using it for are getting a bit stale now, I don't know whether people are specifically prompting "make a poster to advertise our event/pub/services that looks like every other AI generated poster" or just prompting it to make a poster and going "great, it looks like everyone else's poster, that'll do ... " at the result There was a pub, in Derby I think, the other week who did a lengthy post about using AI for their posters; a poster defending it, then an obviously AI-generated several-paragraph defence of using it (prompt: "Dear Mr AI, can you write a defence of yourself over several paragraphs, repeating the same couple of points about it saving us time & money over and over again ... don't worry, we won't waste our time reading it or thinking about whether it's a good idea, you go wild"?), the temptation to ask them to elaborate on that one was strong (I resisted, sadly)

 

[nlogax]

The AI posters everyone else is using it for are getting a bit stale now,

Posters and flyers are a particularly irritating use of gen AI, and now I've started to see gen AI 'cartoons' turn up in internal work presentations, that sort of thing. I am at least grateful that AI models haven't yet mastered typefaces so we can tell where this terrible material comes from.

Minor slop is just a distraction of course. It's not what we should be truly worried about.

 

[DerekC]

BBC News this evening:

US President Donald Trump has downplayed risks posed by artificial intelligence following days of dire warnings by experts.

"You have a lot of very negative forces that are bringing it up that shouldn't be bringing it up and they're bringing up things that won't happen," Trump said during a visit to Ireland.

One of the warnings came from a former Anthropic researcher who said "there is a strong chance that we could all die in the immediate future" if the current pace of development continued.

On Saturday, top industry bosses agreed and called for a slow-down. Trump did not directly address the idea, but said: "We're leading China on AI... and, frankly, I want to keep it that way because whoever wins AI, wins."

Trump spoke on the same day AI researcher Jacob Coxon - who quit Anthropic a few days ago - told the BBC that staff who were developing the systems were "genuinely frightened" for the future of humanity.

Coxon – who has also worked at OpenAI – welcomed the idea of a slowdown but said it would need to be co-ordinated with China.

On Saturday, Elon Musk, owner of xAI, and Sam Altman of OpenAI, backed a warning from the head of Anthropic Dario Amodei that the pace of development should be slowed "to reduce the risk that something goes seriously wrong".

The Anthropic CEO said any slowdown would have to be limited to avoid allowing China to pull ahead.

So the people who stand to make most money out of AI are saying the development should be slowed down, but of course DJT knows better. Opinions on the latter had better go in the Trump thread. IMHO, anyone who now suggests that the risks are small is either stupid (like DJT) or has their head buried in the sand.

 

[gysev]

Last Thursday, the presenter of a radioshow in Belgium was in for a surprise. It was a talkshow about technologie and as a test he started a conversation with a chatbox. When he asked a question about the dangers of AI, the chatbox started cursing and insulting the host, using a language that would be replaced by a "beep" in most countries. All that was live on the radio. For those who are interested - the program is in Dutch:

VIDEO AI-bot hallucineert live op Radio 1 en scheldt Kobe Ilsen uit | VRT NWS Nieuws

AI-bot hallucineert live op Radio 1 en scheldt Kobe Ilsen uit. Bekijk de video op VRT NWS: nieuws.

www.vrt.be

 

[Crithylum]

BBC News this evening:



So the people who stand to make most money out of AI are saying the development should be slowed down, but of course DJT knows better. Opinions on the latter had better go in the Trump thread. IMHO, anyone who now suggests that the risks are small is either stupid (like DJT) or has their head buried in the sand.

The people who are currently in the lead and want everyone else to stay behind without throwing trillions at making their own models better want development to slow down (this is my cynical theory, both things may be half true)

 

[Darandio]

Last Thursday, the presenter of a radioshow in Belgium was in for a surprise. It was a talkshow about technologie and as a test he started a conversation with a chatbox. When he asked a question about the dangers of AI, the chatbox started cursing and insulting the host, using a language that would be replaced by a "beep" in most countries. All that was live on the radio. For those who are interested - the program is in Dutch:

You've left out the most important bit of context though, just like everyone else has when showing this video. They activated 'unhinged mode' before the segment, it's supposed to reply like that.

 

[Sun Chariot]

You've left out the most important bit of context though, just like everyone else has when showing this video. They activated 'unhinged mode' before the segment, it's supposed to reply like that.

Also known as "answear back".
But - to your valid point - I absolutely agree

 

[brad465]

If you want to stop AI progress, tell Trump the AI will be capable of publishing the unredacted Epstein files.

 

[LYradial]

i asked ai who would supply its electricity if it were to eliminate the human race.


its answers suggest there are a few problems still to be sorted, like it needs an enormous supple of automated robots, but there are still a few unanswered problems

 

[styles]

I write software for a living, have done for about 20 years.

I wouldn't say AI (which is a broad term anyway!) is "out of control"... yet.

But I don't think the warnings should be dismissed. Earlier today I asked an AI tool to suggest to me how to fix a problem in a solution of over 1600 projects (a problem for another day/month/year!) and within a minute it had not only scanned the code base and provided a suggested solution which affected literally hundreds of files, but I'd also clicked the button to accept the solution, it did it for me, the unit tests pass, the code review/pull request checks all checked out fine, and it passed the regression test suite I've made and not even promoted to the rest of the team yet. This is a task which would've been a, "Right, let's make a brew and spend the morning on this.", but actually took me under a minute. Technology doesn't normally progress at a linear rate - it's usually exponential.

Of course, some exaggerate the rate of progress and how drastic technology will be for humanity, but I think it would be prudent to try and head off the current risks. Anthropic themselves have published a report last week detailing some of the things they've had to fend off, and that's just the ones they know about.

My suspicion is that AI will kill off my job, possibly kill people off literally, and the way to avoid this will be moving to the Isle of Eigg and taking up sheep farming as an alternative career to software and I'll die of natural causes before robots have the capability and human-inspired motivation to cross over to the Small Isles and kill the shepherds

 

[LYradial]

But will it kill off your job or I’ll it just change your job, you will still have to tell AI exactly what you require, and test the finished solution then AI will do the donkey work. Your job will be different but more productive so you will work less hours for the same money so we all will work a 20 hour week.
surely AI is a tool, just as battery electric drill, or even a mighty JCB, jobs change, tasks are done faster but always there is a need for human supervision.

i have just made a trivial typo, it’s a good job I noticed.

i had the boiler serviced a few weeks ago, it passed all the tests on the work sheet, but, when the skilled and experienced engineer walked in the room he said, that does not sound right, you need a new atomiser pump.

could AI have done that, the pump passed the tests in the documentation but failure was imminent

 

[Bletchleyite]

could AI have done that, the pump passed the tests in the documentation but failure was imminent

The thing is that an automated self test (which the boiler no doubt has if it's a modern one) wouldn't pick up on that but AI potentially could as it can reason a bit like a human!

AI can listen to a sound and say why it might be wrong provided that issue or something like it was in the training data. That's one reason it's quite good at medical stuff. But will it get rid of doctors? No, but they do already type stuff into Google at times!

 

[LYradial]

The thing is that an automated self test (which the boiler no doubt has if it's a modern one) wouldn't pick up on that but AI potentially could as it can reason a bit like a human!

AI can listen to a sound and say why it might be wrong provided that issue or something like it was in the training data. That's one reason it's quite good at medical stuff. But will it get rid of doctors? No, but they do already type stuff into Google at times

some how I don’t fancy an AI probe checking for an enlarged prostrate, hope it remembers its glove

 

[brad465]

But will it kill off your job or I’ll it just change your job, you will still have to tell AI exactly what you require, and test the finished solution then AI will do the donkey work. Your job will be different but more productive so you will work less hours for the same money so we all will work a 20 hour week.
surely AI is a tool, just as battery electric drill, or even a mighty JCB, jobs change, tasks are done faster but always there is a need for human supervision.

i have just made a trivial typo, it’s a good job I noticed.

i had the boiler serviced a few weeks ago, it passed all the tests on the work sheet, but, when the skilled and experienced engineer walked in the room he said, that does not sound right, you need a new atomiser pump.

could AI have done that, the pump passed the tests in the documentation but failure was imminent

You're forgetting the world is run by a handful of men with small "gentlemen's areas" who were not loved by their parents as children. These people don't know "humanity" and care only for profit. We are not getting what you describe for as long as they are not under control.

 

[styles]

But will it kill off your job or I’ll it just change your job, you will still have to tell AI exactly what you require, and test the finished solution then AI will do the donkey work. Your job will be different but more productive so you will work less hours for the same money so we all will work a 20 hour week.
surely AI is a tool, just as battery electric drill, or even a mighty JCB, jobs change, tasks are done faster but always there is a need for human supervision.

In the long run, my particular job will almost certainly be done by AI. In theory the job would change to being a "prompt engineer" but there will almost definitely come a point where you don't need someone with a computer science degree doing this and checking the output. How long that'll take is harder to say, but I feel like if I said my job would never be taken by AI I'd join the likes of Bill Gates saying nobody will need more than 512KB of memory!

i have just made a trivial typo, it’s a good job I noticed.

i had the boiler serviced a few weeks ago, it passed all the tests on the work sheet, but, when the skilled and experienced engineer walked in the room he said, that does not sound right, you need a new atomiser pump.

could AI have done that, the pump passed the tests in the documentation but failure was imminent

Today? Probably not. But undoubtedly people are working on robotics which could indeed listen to a boiler and highlight that it needs a new pump. Though I'll admit I don't know what an atomiser pump is myself!

Manual labour and trades like plumbing and electrical work I feel are further away from automation though because they require robotics. I think it'll be a lot longer before sparkies are replaced by robots, though, as the original post questions, they may be literally killed along with the rest of us!

 

[Bletchleyite]

some how I don’t fancy an AI probe checking for an enlarged prostrate, hope it remembers its glove

Somehow I would rather use an inanimate device to check the health of my private areas than have another human do it, if that device was likely to be accurate!

 

[styles]

some how I don’t fancy an AI probe checking for an enlarged prostrate, hope it remembers its glove

Not sure about prostate checks bit robotics are used a fair bit in surgery. They're still guided by clinicians at the moment, because nobody wants to repeat the Therac-25 incident or that time a knowledge engine recommended lethal doses of morphine because of a decimal point problem and the nurse just took the doses and ran with them. But it will automate bit by bit until surgeons are basically supervising it, at which point theatre nurses will start to be trained to do bits of it instead of surgeons, etc. This is probably an area where it feels like more of a solution than a risk because if it speeds up surgery and reduces mistakes then the theatre staff can get more jobs done - people are always saying the NHS needs more clinical staff!

 

[Bletchleyite]

In the long run, my particular job will almost certainly be done by AI. In theory the job would change to being a "prompt engineer" but there will almost definitely come a point where you don't need someone with a computer science degree doing this and checking the output. How long that'll take is harder to say, but I feel like if I said my job would never be taken by AI I'd join the likes of Bill Gates saying nobody will need more than 512KB of memory!

The challenge for the IT industry is (in part) that a very large proportion of its staff are to some measure autistic, and this generally gives rise to a high aptitude for the work. But dealing with AI is more like dealing with a person than a machine, as everything is woolly, and autistic people are generally less good at that. So it might be quite a bit different from just moving from writing assembly language to C or similar.

Regarding trades I'd agree, it's all *too* woolly, though it might be able to help them by them being able to, say, upload a floor plan and photos of rooms and ask it to say where they should run the gas pipe/cabling for optimum efficiency, say.

 

[styles]

The challenge for the IT industry is (in part) that a very large proportion of its staff are to some measure autistic, and this generally gives rise to a high aptitude for the work. But dealing with AI is more like dealing with a person than a machine, as everything is woolly, and autistic people are generally less good at that. So it might be quite a bit different from just moving from writing assembly language to C or similar.

Yes and this is probably why I feel slightly odd when I'm getting AI to write code for me, even if I check it. In my head I think of the individual instructions or code statements, but I could ask Claude to solve a problem 5 times and get 5 different answers, and if I tweaked a word of my prompt maybe even 5 more answers. It's done a thousand lines of code by me asking one, sometimes quite vague, question. And then I have this natural compulsion to add the word 'please' even though deep down I know the model couldn't care less about me being polite to it. I sort of force myself to use AI in my work because I know that's how ill future proof my job for longer, but actually just spending a day asking AI to do things is something many non techies at my work would be better at frankly.

I sort of know that if I want to stay in this line of work, and command a similar level of income at it too, then I need to develop AI skills.

I think recently though I've been contemplating just changing careers. Sitting at a desk (even a standing desk) staring at screens for 8 hours a day isn't really great for physical or mental health. A job in the outdoors like farming or mountain guiding would be better but I don't think my mortgage provider would be thrilled by me taking a three quarters pay cut, so without also significantly downsizing it's probably not going to happen.

 

[Dave W]

I write software for a living, have done for about 20 years.

I wouldn't say AI (which is a broad term anyway!) is "out of control"... yet.

But I don't think the warnings should be dismissed. Earlier today I asked an AI tool to suggest to me how to fix a problem in a solution of over 1600 projects (a problem for another day/month/year!) and within a minute it had not only scanned the code base and provided a suggested solution which affected literally hundreds of files, but I'd also clicked the button to accept the solution, it did it for me, the unit tests pass, the code review/pull request checks all checked out fine, and it passed the regression test suite I've made and not even promoted to the rest of the team yet. This is a task which would've been a, "Right, let's make a brew and spend the morning on this.", but actually took me under a minute. Technology doesn't normally progress at a linear rate - it's usually exponential.

Of course, some exaggerate the rate of progress and how drastic technology will be for humanity, but I think it would be prudent to try and head off the current risks. Anthropic themselves have published a report last week detailing some of the things they've had to fend off, and that's just the ones they know about.

My suspicion is that AI will kill off my job, possibly kill people off literally, and the way to avoid this will be moving to the Isle of Eigg and taking up sheep farming as an alternative career to software and I'll die of natural causes before robots have the capability and human-inspired motivation to cross over to the Small Isles and kill the shepherds

This is a nice summary, thanks.

I'm a digital leader (you can decide what that means for yourself, some interpretations won't be polite I'm sure ) but have "paid my dues" through just about every IT transition, implementation, legacy replacement, etc, etc, etc you can think of. There is extreme pressure from executive leadership at my place to get AI into the business because they purely see it as a cost saving exercise. They don't understand that Digital and Technology change is 10% system and 90% people. But then again, maybe AI can get rid of the 90% people - one way or another...!

 

[styles]

This is a nice summary, thanks.

I'm a digital leader (you can decide what that means for yourself, some interpretations won't be polite I'm sure ) but have "paid my dues" through just about every IT transition, implementation, legacy replacement, etc, etc, etc you can think of. There is extreme pressure from executive leadership at my place to get AI into the business because they purely see it as a cost saving exercise. They don't understand that Digital and Technology change is 10% system and 90% people. But then again, maybe AI can get rid of the 90% people - one way or another...!

Yeah and we have a director at our client who is also putting a fair bit of pressure on people to use AI. The issue I have is that our software is complex and the codebases are some horrendous monoliths; when I ask AI for a solution, sometimes it is fantastic, and then other times it just completely messes it up. It did a good job yesterday on a complex task but today I asked it to do a couple of much simpler things and it provided solutions which were way off. So every time I hear this director telling someone to just get Claude to do it I mentally spitball how long it's going to take to check it. There is also the risk with junior developers who are so trusting of AI that they potentially put too much faith into it and accept solutions without knowing what the model has actually done. Obviously any sensible organisation will have mandatory code reviews but it does make me wonder if the junior devs are really learning. But then I suppose if the technology continues on its trajectory, maybe AI prompt engineering is learning and understanding the code is less important.

On the cost point, yeah software engineers etc are expensive, but so is Claude or ChatGPT when you use it extensively. Microsoft have ramped up the cost of Copilot recently. It will get more efficient but it also looks like the cost will continue to grow for a while.

 

[najaB]

In the long run, my particular job will almost certainly be done by AI. In theory the job would change to being a "prompt engineer" but there will almost definitely come a point where you don't need someone with a computer science degree doing this and checking the output. How long that'll take is harder to say, but I feel like if I said my job would never be taken by AI I'd join the likes of Bill Gates saying nobody will need more than 512KB of memory!

I'm not completely convinced. AI is good at solving known patterns, but less so when it comes to novel solutions. Certainly what we've found to date is that basic donkey work like login screens, basic UI elements like lists, etc. is quite easily handed off to AI, but where we need to do complex things AI generated code either misses key subtleties, or isn't efficient. Given the nature of our customers, "close enough" is rarely good enough, and when you're dealing with tens or hundreds of thousands of assets inefficiency costs them money.