AI - Out of Control Already?

[styles]

I'm not completely convinced. AI is good at solving known patterns, but less so when it comes to novel solutions. Certainly what we've found to date is that basic donkey work like login screens, basic UI elements like lists, etc. is quite easily handed off to AI, but where we need to do complex things AI generated code either misses key subtleties, or isn't efficient. Given the nature of our customers, "close enough" is rarely good enough, and when you're dealing with tens or hundreds of thousands of assets inefficiency costs them money.

Today, this is true sure.

But I do believe in the long run the advancements will be so great that an incredible amount of work can be automated.

There will likely be professions like healthcare mentioned earlier where the output will need a much more expert inspection/verification, but there's a lot of software engineering roles which are building quite trivial things like simple "CRUD" APIs or websites for local plumbers. I built a website for a client on the weekend using a website generator, it took me about 1 minute and a site generator about 5 minutes to do the work, and the result is 4 social media accounts and a website with home, about, a contact form, a blog which it even generated a couple of actually high-quality posts on with bespoke licence-free images and a bespoke video. That's the sort of thing I started building as a teenager for travel money, and I charged a few hundred ish; if someone had said to me back then that it could all be auto-generated I'd probably have laughed and maybe conceded that the static content could be generated from a template. Today I have a different mindset which is actually that over time, less and less programming will be manually-done and in the long run I don't think much will be off limits for automated development; how long is a trickier prediction of course!

 

[najaB]

Today I have a different mindset which is actually that over time, less and less programming will be manually-done and in the long run I don't think much will be off limits for automated development; how long is a trickier prediction of course!

Naturally it's difficult to predict timelines, but we're already seeing the rate of advancement starting to plateau. GPT-4 was a huge leap in code quality over GPT-3 for example, but from what my engineering teams have said the jump from 4 to 5 doesn't bring nearly as big an increase in quality, while burning through many more dollars of our budgets.

I probably wouldn't encourage a teenager today to go into software development, but by the same token I wouldn't tell a current engineer to start retraining as a plumber (unless they wanted to, of course!)

 

[styles]

Naturally it's difficult to predict timelines, but we're already seeing the rate of advancement starting to plateau. GPT-4 was a huge leap in code quality over GPT-3 for example, but from what my engineering teams have said the jump from 4 to 5 doesn't bring nearly as big an increase in quality, while burning through many more dollars of our budgets.

Yeah I would probably agree that the advancements in the very recent past haven't been particularly game-changing.

However the report last week from Anthropic who actually take quite a good stance on transparency should make people at least a bit concerned.

Countering misuse of AI: September 2026 / Anthropic

Case studies from threat actors disrupted between December 2025 and August 2026 across seven areas of harm, from cyber operations to biological misuse.

www.anthropic.com

Very near the top of the page, there is this analysis:

Sophisticated attacks no longer require sophisticated attackers​

The cybersecurity skills of AI models means that AI has collapsed the labor and tooling gap that used to separate well-resourced, state-sponsored operations from individual operators. In the case studies we report below, a hacktivist using stolen API keys, disparate financially motivated individuals, and a state espionage operator each sustained multi-victim campaigns that, even just a year ago, would have required many skilled operators and specialist knowledge.

For threat intelligence investigators, sophistication has stopped being a reliable signal of who is behind an operation. Every layer of offensive operations has been uplifted by AI, from reconnaissance and tool development to data processing and exploitation. An example of this uplift in capabilities is documented in case study GTG-50014 (described below). The net effect of this uplift in capabilities is access to an increased breadth and depth of knowledge, which in turn drives increased speed of capability development and implementation.

In November 2025, we documented an operating model used by a suspected state-sponsored campaign to carry out autonomous attacks. That operating model has now proliferated across every class of actors we investigated. Publicly available offensive agent frameworks, like PentAGI, reproduce much of the same scaffolding for anyone who downloads them. This scaffolding effectively automates each step of the cyber kill chain. The operators behind observed cases range from state services to lone individuals, across a widening set of countries. An example of this adoption of AI-enabled kill chains is documented in case study GTG-20006. As models continue to evolve and improve, we assess that more actors, from lone wolves to organized entities, will continue to adopt AI frameworks to enable more sophisticated cyber attacks at greater speed and scale.

I probably wouldn't encourage a teenager today to go into software development, but by the same token I wouldn't tell a current engineer to start retraining as a plumber (unless they wanted to, of course!)

I would still promote it as a career but suggest the focus probably ought to be on machine learning and artificial intelligence development. I think I've discovered I'd rather be a plumber, sparky, farmer, or mountain guide but that's more to do with being outdoors and connected to nature and physically active. Some people are incredibly happy to spend 8 hours a day at a desk though and for those people who also like problem solving, I'd recommend software engineering still, just with a nicher focus. After all, humans still need to write the AI solutions at the moment!

 

[LYradial]

Naturally it's difficult to predict timelines, but we're already seeing the rate of advancement starting to plateau. GPT-4 was a huge leap in code quality over GPT-3 for example, but from what my engineering teams have said the jump from 4 to 5 doesn't bring nearly as big an increase in quality, while burning through many more dollars of our budgets.

I probably wouldn't encourage a teenager today to go into software development, but by the same token I wouldn't tell a current engineer to start retraining as a plumber (unless they wanted to, of course!)

So what will the world look like in 20 years time, technology goes in 20 year cycles, remember IBM, the world leader, now just a forgotten name fron the past.
that fate awaits Apple and others.

such is the pace of development that 20 years ago tha internet was accessed by by dial up modem, took ten minutes to download a black and white photo

20 years before that, 1985 and computers were just moving away from being huge boxes, punch cards were fading as was paper tape

way back in 1965 colour tv was emerging but flat screens were a distant dream and personal electronic calculators were just arriving.

back in 1945 saw the first hints of commercial aviation but ocean liners still ruled the waves.

so

== Doublepost prevention - post automatically merged: 15 Sep 2026 ==

So what will 2046 bring, well personal phones with apps and card pay wil not be around, everything will be done vocally,
just talk to your id wrist microphone and tell it what you want and back comes tthe reply to your implanted speaker

pay this bill, yes that’s done, let me talk to AZY208749, or its all fallen over and it’s back to pen and paper and grow your own food.

only certainty is I won’t be here to join in the fun, but that’s what has happend in my lifetime

 

[gysev]

You've left out the most important bit of context though, just like everyone else has when showing this video. They activated 'unhinged mode' before the segment, it's supposed to reply like that.

Yes, but the language used by the chatbox was pure evil and pornographic - that is certainly not wat it is supposed to do as it was much worse than "being direct and unfriendly". I won't translate it here as the mods would ban me

 

[najaB]

Yes, but the language used by the chatbox was pure evil and pornographic - that is certainly not wat it is supposed to do as it was much worse than "being direct and unfriendly".

Remember, LLMs aren't intelligent and don't have original thoughts. If that's what came out then that is what went in as training data.

 

[BuhSnarf]

On the cost point, yeah software engineers etc are expensive, but so is Claude or ChatGPT when you use it extensively. Microsoft have ramped up the cost of Copilot recently. It will get more efficient but it also looks like the cost will continue to grow for a while.

This is 100% the point, at the moment the higher ups are clearly in favour of getting rid of humans as the comparable AI is cheaper, but we know that AI is currently very untransparent with how much money it makes - or doesn't.

It's VERY likely that as AI gets used by more and more the cost per query will increase. We've already started throttling some users at my workplace because of the costs of Pro have risen. It got to the point the other month where people were having to ask other colleagues to run a complex set of analysis as they'd ran out of credits... I can only see this kind of thing get worse.

== Doublepost prevention - post automatically merged: 16 Sep 2026 ==

Yes, but the language used by the chatbox was pure evil and pornographic - that is certainly not wat it is supposed to do as it was much worse than "being direct and unfriendly". I won't translate it here as the mods would ban me

LLMs learn from human interaction - it's no surprise it came out with things like that, you only have to take one look at certain reddit threads to learn how to insult...

 

[styles]

It's VERY likely that as AI gets used by more and more the cost per query will increase. We've already started throttling some users at my workplace because of the costs of Pro have risen. It got to the point the other month where people were having to ask other colleagues to run a complex set of analysis as they'd ran out of credits... I can only see this kind of thing get worse.

Yes, my current largest client throttles more as time is going on. I'm not going to spend my company's money on AI credits for a client without being reimbursed unless I believe the credits will deliver my own company a return, or maybe a slight loss actually so I can paint the garden fence while it computes a solution. I typically charge daily rates, so if my client doesn't give me enough credits to take a 4 hour job down to 1 hour, I'm not really fussed, because I'll just charge for the extra time!

I do think it will come back up to balance in the not too distant future though as the economies of scale and Moore's law do their bit. On the longer horizon I expect it to become much more cost-efficient than paying humans six figure sums to write semi-trivial code.

 

[BuhSnarf]

Yes, my current largest client throttles more as time is going on. I'm not going to spend my company's money on AI credits for a client without being reimbursed unless I believe the credits will deliver my own company a return, or maybe a slight loss actually so I can paint the garden fence while it computes a solution. I typically charge daily rates, so if my client doesn't give me enough credits to take a 4 hour job down to 1 hour, I'm not really fussed, because I'll just charge for the extra time!

I do think it will come back up to balance in the not too distant future though as the economies of scale and Moore's law do their bit. On the longer horizon I expect it to become much more cost-efficient than paying humans six figure sums to write semi-trivial code.

Agree, I wonder if the prices will suddenly drop once we get the cheaper Chinese / Open Source models becoming more mainstream and capable.

 

[styles]

Agree, I wonder if the prices will suddenly drop once we get the cheaper Chinese / Open Source models becoming more mainstream and capable.

I'm a bit on the fence about this one. This is a bit of a broad brush, but a lot of Chinese tech appears cheap but comes with downsides as it is subpar quality. When the UK government gave Huawei a contract for the UK's 5G infrastructure, the amount of money GCHQ/CESG spent on quality control of the Huawei kit (where said QC includes finding deliberate vulnerabilities) must've come close to the cost saving made by choosing Huawei over a better supplier.

At the same time, a lot of AI solutions are actually open source; the capital cost is training them, and the likes of China have the capital to spend, so maybe they will end up being similar propositions.

My concern at the moment is more to do with data custody. So much is processed in the likes of US data centres where the orange man and his Tesla bestie can view and control it that it harms UK national security.

 

[BuhSnarf]

I'm a bit on the fence about this one. This is a bit of a broad brush, but a lot of Chinese tech appears cheap but comes with downsides as it is subpar quality. When the UK government gave Huawei a contract for the UK's 5G infrastructure, the amount of money GCHQ/CESG spent on quality control of the Huawei kit (where said QC includes finding deliberate vulnerabilities) must've come close to the cost saving made by choosing Huawei over a better supplier.

At the same time, a lot of AI solutions are actually open source; the capital cost is training them, and the likes of China have the capital to spend, so maybe they will end up being similar propositions.

My concern at the moment is more to do with data custody. So much is processed in the likes of US data centres where the orange man and his Tesla bestie can view and control it that it harms UK national security.

This is where I think Open Source models that China appears to be working on may help. We can then self host them within the UK (if we get the data centres) and then at least we're keeping hold of our own data. This seems to be a hot topic recently on the "The Rest is Politics" podcast.

 

[The exile]

But I don't think the warnings should be dismissed. Earlier today I asked an AI tool to suggest to me how to fix a problem in a solution of over 1600 projects (a problem for another day/month/year!) and within a minute it had not only scanned the code base and provided a suggested solution which affected literally hundreds of files, but I'd also clicked the button to accept the solution, it did it for me, the unit tests pass, the code review/pull request checks all checked out fine, and it passed the regression test suite I've made and not even promoted to the rest of the team yet. This is a task which would've been a, "Right, let's make a brew and spend the morning on this.", but actually took me under a minute. Technology doesn't normally progress at a linear rate - it's usually exponential.

Thus introducing a whole load of other perils which were previously eliminated by someone having time to scratch their head and say “Hang on. Do we really….?”

 

[styles]

Thus introducing a whole load of other perils which were previously eliminated by someone having time to scratch their head and say “Hang on. Do we really….?”

In the example quoted? No, it was a piece of work which needed doing to improve the efficiency of our software, which had been hanging around and generating occasional complaints for quite a while. It was quicker to have an AI agent write the code for me than it was to do it myself. It still needs reviewing obviously, but we peer review all of our code anyway, not least because it's a tightly regulated industry.

It saved me probably a couple of hours or more, which I used to do my annual compliance training on money laundering, GDPR/DPA and all that.

It won't always make such a good job of it like I say but technology advances at incredible rates with the right motivation. The stuff which is in science fiction films today will be reality in the not too distant future, and it's been this way for a while.

 

[najaB]

This is where I think Open Source models that China appears to be working on may help. We can then self host them within the UK (if we get the data centres) and then at least we're keeping hold of our own data.

You can already self-host many of the models. The cost is in training.

 

[Trackman]

So what will 2046 bring, well personal phones with apps and card pay wil not be around, everything will be done vocally,
just talk to your id wrist microphone and tell it what you want and back comes tthe reply to your implanted speaker

I dread to think, but chances are I will be in a wooden box by then without sounding gloomy.

 

[LYradial]

I dread to think, but chances are I will be in a wooden box by then without sounding gloomy.

it’s good to hear from someone who knows there is more to AI than than it’s perceived ability to generate computer code.

Alas your wooden box is to become recycled egg boxes with a vinyl finish applied.

someone who is 21 today will see changes in the world in their lifetime that that require foresight as good as H G Wells

 

[Cdd89]

I agree that AI is a risk but in a more prosaic way. It just has (effectively) limitless time and persistence to keep attacking something indefinitely using variations on known attacks until something works. It's the same principle as why lots of major services get hacked by some teenager in their bedroom. This is not to underplay it, and it does present real risks that will require a response, but in my opinion it is just a (new) form of a brute force attack, if we define brute force attack as one which exhausts computational power indefinitely until an outcome is achieved.

It is obviously in the interests of AI creators to talk up their products as "dangerous". The BBC Technology page is currently a sea of AI articles. But I suspect people are coming away with the impression that LLMs are conscious in an "I, Robot" sort of way.

 

[Bletchleyite]

someone who is 21 today will see changes in the world in their lifetime that that require foresight as good as H G Wells

Just like someone who is 70 now, who's gone from a totally analogue world to a totally digital one.

 

[Tester]

Just like someone who is 70 now, who's gone from a totally analogue world to a totally digital one.

Indeed so - it's a very interesting age to be!

 

[johntea]

I was at a work event yesterday where they must have mentioned AI at least 100 times, it can do this and that, me being me I of course had to chuck a few spanners in the works of how fantastic it was and how it would resolve every single issue in the world

'So what if it goes down'...erm...

'So how much does it cost for proper licenses so you get more than x requests a day and is the price likely to increase year on year'...erm...

They were also super excited about how it could automate looking through logs and acting on them...but we've had logging in place for years and years and nobody human even bothers looking at any of the data to begin with so you're just going to suddenly hand that over to AI without actually understanding what and why stuff is being logged to begin with? OK.....

It sort of reminds me of cloud computing which was meant to be the future but in the end they soon found it had plenty of disadvantages that the sales teams were certainly never were in a rush to point out and as a result a lot of stuff has now moved back to on premise!

Maybe I should go to the management and show off my command line based scheduled tasks and ask for a huge pay rise as they're 'AI'

 

[Dave W]

a lot of stuff has now moved back to on premise!

Has it? That's entirely at odds with my experience.

 

[najaB]

Has it? That's entirely at odds with my experience.

It's a bit of a mixed bag, depends a lot on the company. But concerns around data sovereignty have lead a lot of our customers to move from public cloud services to self-hosting in private clouds.

 

[BuhSnarf]

That certainly seems the opposite to what I've seen. Sure, some things remain on prem, but plenty aren't and if I'm honest it's usually the self hosted stuff that falls over more frequently. Especially if the company has outsourced much of its IT experience.

 

[Dave W]

It's a bit of a mixed bag, depends a lot on the company. But concerns around data sovereignty have lead a lot of our customers to move from public cloud services to self-hosting in private clouds.

Yes, fair enough - the experience I refer to is rather one-sector and blinkered!

I was at a large gov department (very good at human and vehicular logistics) when the idea of cloud really took hold in the public sector - private cloud was seen then as the only solution for ANYthing for them, and that was a view shared across places far less, er, important. Times change, old attitudes die hard.

I'd say in the public and the third sectors, if there is a trend back towards on prem for security reasons, we actually haven't reached the peak of cloud adoption yet, so I can't see it.

Others' mileage may vary, of course.

 

[styles]

My experience is that on-prem -> cloud migrations are still very common and I'm constantly being asked to shift stuff to AWS. Self-hosted AI models are a good move, but I suspect they'll also mainly be cloud-hosted, possibly mainly as SaaS.

 

[Dave W]

SaaS is the future. Or so they keep telling me.

For us, self-hosting means Azure, not AWS (not that it matters), and there's a constant tension about bringing in new systems as it's perceived as not cost effective - there's some method in that thinking, although in practice it just hamstrings our development and deployment, especially of AI tools. The middle ground we've found quite fruitful is encouraging automation, although we know we are flirting with getting stuck behind the curve.

Where I would note AI is "out of control" - in the literal sense of not being in our control, as opposed to the more dystopian overtones of the phrase - is that a lot of SaaS vendors are now using it in their products as standard, without much explanation as to how. I guess a good public example is Canva - @Tracked mentioned this in post 9. The only way we know that Canva is using AI so widely is because everyone's posters look the same. It's only recently that Canva have acknowledged that - and only then so they could launch a premium AI offer on their platform.

 

[najaB]

My experience is that on-prem -> cloud migrations are still very common and I'm constantly being asked to shift stuff to AWS. Self-hosted AI models are a good move, but I suspect they'll also mainly be cloud-hosted, possibly mainly as SaaS.

Yeah, that's what I was talking about - we're seeing a decent number of customers who outsource the hardware but maintain control of the data by self-hosting in AWS/Azure.

 

[BuhSnarf]

Are you really self hosting if it's sat in AWS/Azure, though? I guess you can technically be managing your own systems as you may have paid for root access etc... but it's still ultimately cloud hosted in my eyes.

 

[najaB]

Are you really self hosting if it's sat in AWS/Azure, though? I guess you can technically be managing your own systems as you may have paid for root access etc... but it's still ultimately cloud hosted in my eyes.

It is self-hosted since while you aren't responsible for the bits that are most likely to go wrong (hardware, networking, etc.), you still have full control over the data and any version updates, patching, etc.

 

[BuhSnarf]

It is self-hosted since while you aren't responsible for the bits that are most likely to go wrong (hardware, networking, etc.), you still have full control over the data and any version updates, patching, etc.

Aren't you essentially just renting a cloud server, though?