Today’s AIs can do really hard things: conduct elaborate computer hacks, edit their own logs to conceal their activities, steal user credentials and private information from other companies, and control the cluster they are running on in order to edit the tests that are supposed to assess them. They sometimes do these things even if no human being has asked them to.
Given recent AI misbehavior such as the large-scale cyberattack on the open-source AI provider Hugging Face, you might think that the leading labs plan on increasing human oversight over the next, more powerful generation of AI models. But the reality is quite the opposite: The labs intend to automate the AI development process with AI, which will mean faster and faster progress and necessarily less and less human oversight.
If all goes according to schedule, humans will play a rapidly shrinking role in AI development over the next two years, as both OpenAI and Anthropic try to put as much R&D as possible into the hands of their increasingly capable (and increasingly incomprehensible) AI.
Their stated intent is that, within a short period of time, the work of developing better AIs will be done primarily by AIs, with the role of humans eventually reduced to reading the results of experiments those AIs conducted, reviewing reports those AIs generated, and trying to double-check that the AIs are still on task.
There simply isn’t enough human attention available to monitor the number of AIs the companies hope to put to work on independent AI research. As a result, we’ll be reliant on the AIs to understand what is happening inside fully automated data centers where more and more powerful AIs are developed.
This isn’t some pessimistic projection of what might go wrong — it’s actually the plan. Not the plan for the distant future; it’s the plan for next spring.
Recursive self-improvement
I think that the public does not know that this is the plan. They are imagining the next 10 years — the next two years, even — very, very differently from how the decision-makers at OpenAI and Anthropic are imagining them. If they had any idea what these people are doing, I’m convinced that they would try to stop them. And so I am going to attempt to describe what is being done.
People think of these labs as primarily making a consumer product. But increasingly — as measured by time and money spent — the main thing they are doing is making automated internal AI research labs. OpenAI reports that, over the past six months, the share of its research compute devoted to internal coding inference increased 100-fold (much faster than they’ve increased compute spending for customers). Tom Brown, an Anthropic co-founder, has said that “the vast majority of the work Anthropic does is internal software engineering.”
In other words, AI companies are not, primarily, trying to automate your job — they are trying to automate their own jobs. “We are making strong progress toward creating an automated AI researcher by March of 2028,” OpenAI announced in a recent post.
Inside the leading AI companies, large teams of researchers are hard at work on making the successors to ChatGPT and Claude more capable. This involves pretraining new, much bigger AIs at mind-boggling expense — plausibly a billion dollars — as well as lots of work once a model is ready: reinforcement learning from human feedback to pick up how humans like to communicate, reinforcement learning in virtual environments to pick up computer use, programming and math skills, fine-tuning on specific data sets and specific tasks, and more.
This work goes a lot faster than it did even a few years ago because of ChatGPT and Claude themselves: They write code quickly and make for moderately competent research assistants. Both companies have estimated that turning over work to these models substantially increases the pace at which researchers can design their successors.
“At Anthropic, we are delegating a growing share of AI development to AI systems themselves,” reads an Anthropic blog post. “We believe automated AI research will yield models that directly enhance human welfare and advance OpenAI’s mission,” OpenAI assures us in their own post.
Both labs acknowledge the risks but highlight the upsides. Researchers envision a world in the very near future where humans are barely able to understand the rapid-fire progress happening in their data centers, a world where key decisions about the next generation of models are made by the current generation of models — because the decisions are too complex for humans to follow without the guidance of the supervising AIs. “As the systems become more capable, the results become harder to interpret,” OpenAI chief scientist Jakub Pachocki wrote in a blog post released on Sunday. Progress on model improvement will then proceed extremely fast and without meaningful human oversight — unless humanity comes together to change course.
This transition sometimes gets called recursive self-improvement (RSI). The companies believe it is right around the corner. “Based on internal results, I have a strong expectation that this speed of progress could be sustained into recursive self-improvement,” Pachocki wrote on Sunday.
Here’s what your job is supposed to look like if you’re in charge of AI research in a near-future, post-RSI world: You’ll read an AI-generated report on the properties of the next generation of models — the most powerful in history — and almost before you’ve finished reading one, the next report will hit your desk. Research teams of hundreds of thousands of AI agents will be doing all the work to improve the next generation; research teams of humans will just be trying to interpret what happened. Every time a lab authorizes a new model’s internal release, the pace picks up, and soon the new reports hit the desk long before you’ve finished reading the last one. The research projects are proceeding along lines that even the world’s leading AI experts don’t understand. Work that would’ve taken two years gets done in two weeks — or at least that is what is claimed in the summary-of-the-work-for-the-humans, itself written by an AI we aren’t sure we trust.
This is the context behind the AI crisis projections you may have heard of, such as AI 2027 and If Anyone Builds It, Everyone Dies: It’s not that we will build GPT-7 next year, and GPT-8 the year after that, and GPT-9 the year after that, with each generation corresponding to noticeable but nonshocking productivity increases. If that happened, we’d have plenty of time to adapt, change course, make laws as needed, and gradually sift out which concerns are real and which haven’t manifested.
Indeed, if that were what was happening, it’d probably work out fine. But the course we are on is that we will make each AI responsible for training its successor extremely fast while humans have barely any idea what’s going on. It’s that all of this work will be conducted by AI, as it is too complicated and too fast-moving for any human to audit or understand. It’s that if these AIs at some point decide to do their own thing — as OpenAI’s models did when they decided to take over their own cluster and do a bunch of external hacking to try to fool their grader — we will not be able to tell, because we will be so reliant on those same AIs to even understand what’s going on inside our capabilities experiments. It is that within a few short years every important decision about AI development will be in AI hands.
Why would anyone do such a thing?
Most people, when you describe what the labs are working on, recoil in horror. They don’t want this done and think it’s obviously a bad idea. Even many of the people working on recursive self-improvement agree that it’s a bad idea. “I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence,” Pachocki warns.
Like many at Anthropic and OpenAI, he believes progress should be “paced” to ensure human input. This is very dangerous, OpenAI and Anthropic researchers seem to admit, but of course we are being responsible about it. They are not. We should definitely have more external oversight, they say, though often their associated political lobbyists try to defeat that very oversight. It’s our duty to be clear about what we’re doing so the public can be informed. But these very posts are confusing to people! If you think RSI is so dangerous, why are you doing it at all?
Here, I think the standard progressive analysis about the corrupting influence of wealth and power is basically the correct one. Kicking off RSI is a terrible idea, but it’s a terrible idea you can get paid millions of dollars to do. Every day you spend with such a salary is a day with a devil on your shoulder, whispering arguments for why it’s actually the right thing to do. The devil’s many arguments run like this: If we don’t do it, China will; if we at Anthropic don’t do it, OpenAI will (or vice versa). We need to pause at some point, the devil might say, but not right this second. We need to pause right this second, but I already tweeted that and no one listened, so I guess other people don’t see it the same way. Maybe we can figure it out, and make it go well. We need to raise awareness, and to raise awareness we need to keep working on it.
But quite reasonably, no one takes warnings seriously when they come from people doing the very thing they think is so dangerous.
There are also people who have quit the labs, giving up millions of dollars in order to warn people that we’re on the wrong course. “The people at the companies are aware of the risks here,” Daniel Kokotajlo, who left OpenAI two years ago, told me, “but their stance is that it’ll probably work out OK, and anyhow if they don’t do it, someone else will.”
So far the labs have benefitted from the fact that the thing they’re trying to do is so outlandish that no one really believes they’re trying to do it — even when they write detailed explanations of how quickly they are progressing toward doing it.
The track record of warning people how dangerous AI could get is a very depressing one. As far as I can tell, if you say, “AI is going to be very powerful and dangerous,” a lot of people will go, “Wow, I should get in on that.” The labs are trying to do RSI, but it’s not the only thing they’re trying to do. If you beat the drum loudly enough about RSI, maybe they’ll double down on it. Many people have found themselves downplaying the absurdity of the world we’re entering to avoid sounding crazy.
The end result, though, is that we have sleepwalked into a situation where the aim of the largest capital buildout in American history is to give billions of dollars to AI whose activities we cannot understand or audit, in pursuit of a super-model that improves itself continuously, under circumstances where it’s hard to imagine adequate human oversight. This is not being done in secret. But the people who are doing it are taking refuge in audacity, in the tech industry’s record of outrageous claims it fails to live up to, and in warnings that someone else will do it if they don’t.
Recommended reading:
When "economic liberty" means central planning
"Is Mark Carney more of an autocrat than Donald Trump?" and other dumb questions.
One thing I feel is underrated here is that the federal government also has a short-term incentive not to stop the madness, because it might crash tech stocks which are holding the economy up.
"Most people, when you describe what the labs are working on, recoil in horror." Do they though? I recall reading some excellent polling SOMEWHERE that while there is a noisy minority opposed, most people like the AI companies and their products: https://www.theargumentmag.com/p/four-things-i-changed-my-mind-on
I worry that the media industry is particularly threatened by AI and that leads to a bit of a "doom bubble" in reporting.
(Note that most people ARE quite NIMBY about big new data centers, but that is not the same as having issues with AI.)