The state and the machine

By Iain, 17 Jun 2026

What little we saw of Fable and Mythos offers both cause for excitement and concern. It was widely and credibly seen as a model of a completely different caliber from those that had come before. Perhaps the risks in this instance were overstated or amplified for political ends. What is more profound is that the short time we had with the models offered a clear glimpse of a future in which a single company is making significant progress toward a superintelligence with the potential to rival or exceed the power of nation-states or even massive corporations. That juncture was never going to arrive smoothly or without consequences.

The letter reached Anthropic at twenty past five on a Friday evening, and by Saturday the company had pulled its two most capable models offline for every customer on the planet. The US Commerce Department had decided that Fable 5 and Mythos 5 were too dangerous to leave in foreign hands, and its order was worded broadly enough that no foreign national could touch them, whether they sat in Shenzhen or three desks away in Anthropic’s own San Francisco office.

So the company shut the models off for everyone, including American customers, and spent the weekend insisting the whole thing was a misunderstanding.

There is a sense of inevitability that we would someday reach this point, but we should focus less on the ban than the thing that got us here. Somewhere in the past year, the industry crossed a line that had once been theoretical: from AI that helps people build AI to AI that continually builds and improves AI. I made the narrow case for this in March, writing about Google’s machine that improves the machine. What follows is the wider version. How a model barely a week old and powerful enough to be classed as a weapon can be switched off overnight by someone who does not work for the company that built it.

I’ll also consider what this means now we have entered a new phase in which these tools have grown powerful enough to count as a matter of state, and nobody has agreed who should be in charge of them.

What happened on Friday

Anthropic says the letter gave no detail of the actual security concern, and that the government’s worry, relayed second-hand, traced back to a jailbreak Amazon had flagged, one that turned Fable 5’s knack for hunting software vulnerabilities into something pointable at the wrong targets. Anthropic argues the flaw was overstated, that the same holes could be found in models already in public hands, and that access to its other systems, Opus 4.8 among them, remains untouched. None of which changed the outcome over a weekend.

Bizarre as it may sound, there is a precedent for locking a piece of pure software inside the country by calling it a weapon. In the 1990s, the United States classified strong encryption as a munition, in the same legal category as missiles and machine guns. Phil Zimmermann, who wrote the email-encryption program PGP, spent three years under federal investigation as a suspected arms dealer for letting his code reach the internet, and a graduate student named Daniel Bernstein had to sue to establish that source code counted as speech.

The campaigners won that one. PGP’s source was carried abroad as a printed book, scanned and recompiled overseas, and by 2000 the controls had collapsed. Code proved too slippery and too useful to fence in. But with the frontier labs few in number, the underlying weights proprietary and the files enormous, models will be much easier to control.

Leaving aside the legalities, the overall timing borders on slapstick. Eight days earlier, on 5 June, Anthropic’s co-founder Jack Clark and its institute lead Marina Favaro had published a long post asking, in effect, for a brake pedal, arguing that the world should keep the option to slow or temporarily pause frontier AI development so that oversight could catch up with models clever enough to design their own successors. This was met with general skepticism, as to date only leaders or laggards seem interested in anyone taking their foot off the gas.

But then a government reached over and yanked a brake of its own. Anthropic spent a week explaining why society might one day need an off-switch for these systems, then someone flipped one, and Anthropic spent the weekend explaining why that particular use was wrong. The control problem these companies keep warning about in the future tense is already here, and the people who built the technology do not (and arguably should not) have the final say on who holds the switch.

From “AI helps” to “AI builds”

Putting the legal drama aside, the trend that got us here is almost prosaic. As of May, Anthropic says that more than four-fifths of the code merged into its own codebase came from Claude, up from low single digits before Claude Code arrived in February 2025, and a typical engineer there now ships roughly eight times as much code per day as they did in 2024. Those are not forecasts but operating numbers, the internal accounts of a company that is, in a fairly literal sense, already building itself.

And the quality of the output has climbed in step with the volume. METR, which measures how long a task an AI can run before it falls over, reckoned in early 2025 that the best models could see through jobs lasting a human engineer under an hour, and by its January update that horizon had stretched into hours, with the doubling time itself dropping from roughly seven months to around four.

The clearest sign that the loop was closing came when the machines started tuning the machines. I wrote in March about AlphaEvolve, Google DeepMind’s evolutionary coding agent, which turned up a scheduling trick that claws back about 0.7% of Google’s worldwide compute, beat a record for multiplying four-by-four matrices that had stood since 1969, and, in the detail that gives the game away, sped up the training of the very Gemini models that drive AlphaEvolve.

In March Andrej Karpathy, a founding member of OpenAI and once Tesla’s head of AI, put out a 630-line program called autoresearch that runs on a single GPU. You write your instructions in plain English in a Markdown file, and an agent edits the training code, runs a five-minute experiment, keeps the change if the model improved and reverts it if not, then loops through the night while you sleep.

Pointed at his own nanochat project, it trimmed the time-to-GPT-2 benchmark from 2.02 hours to 1.80, an 11% gain on a codebase Karpathy had already spent years hand-tuning. When Shopify’s chief executive Tobias Lütke set it loose on his company’s own data overnight, he woke to a model 19% better than the one he went to bed with. The self-improving loop is no longer a secret kept inside one lab. It fits on a weekend and a single graphics card.

Jack Clark puts the odds of a model building its successor with no human in the loop at around 60% by the end of 2028. Quarrel with the timing if you like. The direction of travel is impossible to argue.

The tipping point

Building an AI is itself a scoreboard problem. Does the next model score higher? Does it train quicker? That makes the work of improving a model exactly the kind the machines have proved good at, which is why the loop from one model to a better one is closing fastest in the spot outsiders can see least, inside the labs, against numbers only they hold.

The reason to watch this rather than the latest benchmarks is that it is a door that opens one way. A modern model is built by code at every level: the data it is fed, the training runs, the synthetic examples it learns from, the reward functions, the harnesses that grade and control it. Once a model is good enough to write and run that code, it can reach into every stage of its own making. The same essay in which Anthropic asks for a pause describes the loop without much drama, with existing models now writing the training recipes and tuning the settings that make the next model. By its own account, the first rung of the ladder is already behind us, and a company turning out four-fifths of its code by machine is not forecasting that future so much as living in it.

A model proposes a change to its own training, runs it against a number, keeps it if the score climbs and bins it if not, then goes round again, through the night, across thousands of experiments at once. None of that is new in kind. What is new is that the thing doing the improving is itself getting better at improving, so each turn of the wheel comes quicker and better than the last. AlphaEvolve sped up the training of the Gemini models that run AlphaEvolve, which is the loop in a single sentence. The human slides from doing the research to managing a swarm of it, and then, by degrees, to watching.

Two things follow. The first is speed. When the improver improves itself, the rate climbs rather than holding, and the METR doubling time falls from seven months to four, which is what an accelerating curve looks like from the outside. Anthropic’s own yardstick is blunter, a jump from models handling four-minute coding tasks in 2024 to twelve-hour ones now.

The second is that the future gets harder to read. Gains drawn from a machine’s search rather than a human’s design arrive jagged and out of sequence, with no roadmap saying which ability shows up when, and the whole thing runs inside a few labs against private numbers, where a leap reaches the rest of us as a finished model rather than a build we watched happen.

What few inadequate controls we have to govern this were built for a slower world. While a model was wrong half the time and could not act on its own, weak safeguards hardly mattered. The means to test a system before release, to know what it can really do, to slow it down or switch it off, to decide who is allowed to run it, all of it could be filed under “later”.

The recursive improvement loop tears up that timetable. Those same controls become vital at the exact moment the thing they govern starts outrunning them, and, for the most part, they are not there. In their absence, two crude instruments are left. One is the government sledgehammer that came down on Fable. The other is the labs marking their own homework. The question has stopped being about software; it is about power, and it has left the people who build these systems in an awkward position.

Nobody is flying the plane

The awkward place belongs, more than to anyone, to Dario Amodei. Nobody has argued harder that this trajectory is coming, and few have pushed harder for someone to regulate it. In an essay published this month, he set out a regime borrowed from the Federal Aviation Administration, under which frontier models, like aircraft, would have to pass testing and auditing before release, judged either by a government body or by private outfits it licenses, with the authority to block or reverse a model that came up short. The incidents he wants logged fall into four areas: cybersecurity, biological weapons, loss of control, and automated research and development.

Even that framework, Amodei warned, might not be enough. A moment may arrive, perhaps sooner than people think, when the most powerful systems stop resembling airplanes and start looking closer to weaponizable nuclear materials, a danger to humanity rather than to public safety alone, calling for something far blunter than anything he had previously put on the table. He published that in June. Within days, the government supplied the blunt instrument, in the form of the Fable ban, addressing a model as precisely the hazard he had described, except that it acted by executive order, over Anthropic’s protests, with the company now arguing the danger was overstated.

As John Gruber observed, Amodei had said in as many words that calling such a moment is not Anthropic’s place to do. The judgment was taken out of the company’s hands before the ink was even dry on Amodei’s words.

What does not exist, anywhere, is the AI FAA Amodei keeps reaching for. No agency tests frontier models, no standing rule says when one should be pulled, no settled process names who decides. Congress has barely stirred. A preliminary bill from two representatives would make developers write risk-reduction plans, Senator Ted Cruz has talked about a measure of his own, and analysts put the chance of anything passing this year below thirty percent, with consumer groups worried that a thin federal law would only pre-empt the stricter rules some states are drafting.

Into that emptiness steps the executive branch, governing by improvisation, and it had a chance to build something sturdier. On 2 June, the administration signed its own AI order, and the notable thing was how little it asked for. Access to new models before release was made voluntary, and the text went out of its way to rule out any mandatory licensing or pre-clearance, the very levers that might have caught something like Fable before it shipped. Having scrapped the previous administration’s compulsory safety testing on day one, this was a deliberate choice to keep things light. Eleven days later the same administration reached for the heaviest tool it owned and switched two of a company’s models off outright.

The two are hard to square. A government that will not set a standing rule for the industry and forbids itself from pre-clearing models does not, obviously, then obliterate one company’s most capable models over a weekend on the grounds of pure safety. The official reason is the jailbreak, and the cyber risk is serious enough that the June order was written partly with it in mind. But the inconsistency invites a less flattering reading.

Earlier this year, the Pentagon designated Anthropic a supply-chain risk and shut Claude out of the department after talks collapsed over how its models might be used for surveillance and autonomous weapons, and Anthropic sued. Set against that, the ban looks less like a safety measure applied without fear or favor and more like the next blow in a quarrel with the one major lab that has refused to bend the knee. Ben Thompson of Stratechery aptly invokes Pericles: you may not be interested in politics, but politics is interested in you.

In the absence of proper laws or guardrails, enforcement ultimately comes down to power. If a frontier model really is nearing the heft of a nuclear weapon, then a private company that means to dictate how the state may use its technology is assembling a rival to the state, and the state clearly will not stand for it.

Satya Nadella has argued vocally that power concentrated in several or perhaps only one all-powerful model has neither political nor societal permission: “There is no…permission for an AI future that hollows out entire industries. Think about what happened in the first phase of globalization where entire industrial economies were hollowed out by outsourcing. The GDP numbers looked fine on the surface, but the displacement was real, and the consequences are still being felt. Let us not bring that dynamic into the AI era, with a small number of AI systems capturing all the economic returns, while entire industries find their knowledge commoditized right out from underneath them.”

Yet globalization happened anyway, and given the current picture, Nadella’s point reads more like a prediction than a warning.

In the United States right now, power rests with whoever holds the White House and treats might-makes-right as the final word, rather than as the thing a constitution exists to check. It also writes off the courts before they have spoken, which is the very wager Anthropic is making by going to law. A handful of private firms are setting the speed of a technology with stakes the size of weapons and economies; those same firms are asking out loud to be governed, and the only tool the state has fielded so far is the power to switch things off.

The East India Company 2.0

There is another way to read the same shutdown, and it points in the opposite direction. Suppose the ban is not the state reining the labs in but the state marking its territory. A government does not move to seize control of something it takes for a toy. By treating Fable 5 as a weapon, the administration confirmed, to Beijing and everyone watching, that American AI firms now sit among the assets a nation guards rather than the products it merely sells.

The historical comparison is the East India Company. A private venture was handed a protected monopoly and a clear run to grow rich and powerful, on the tacit condition, seldom written down, that it served the crown’s strategic ends first and foremost. The charter was the price of the monopoly, and for the better part of two centuries the arrangement suited both sides.

The modern version is an oligopoly and is further along than it looks. The Pentagon has already folded Anthropic, Google, OpenAI and others into its own plans, the AI Now Institute argues the leading labs are being positioned as too big to fail and subsidized as a national strategic priority, and when OpenAI’s finance chief mused last winter about the government backstopping the debt behind the data-center build-out, the suggestion was loud enough to force a White House denial.

A company the state cannot afford to let fail borrows more cheaply, because lenders price in the rescue everyone swears will never come but obviously will. Cheaper money does not slow a build-out; it speeds one. So a ban that looks, on the surface, like the brakes coming on may, underneath, be the accelerator, lowering the cost of capital that buys the next round of compute and, with it, the next turn of the loop.

None of this is settled, but with Fable 5 it feels like we have crossed the Rubicon. We are now in the world of self-improving models so capable that the political and power dynamics that have always threatened to be attendant have arrived.

What little we saw of Fable and Mythos offers both cause for excitement and concern. It was widely and credibly seen as a model of a completely different caliber from those that had come before. Perhaps the risks in this instance were overstated or amplified for political ends. What is more profound is that the short time we had with the models offered a clear glimpse of a future in which a single company is making significant progress toward a superintelligence with the potential to rival or exceed the power of nation-states or even massive corporations. That juncture was never going to arrive smoothly or without consequences.