The Doomer’s Dividend

Second Level Thinking · Issue 52

The frontier labs are warning us that AI might end the world and asking for it to be slowed down. Follow the incentives, and the safest proposal looks like the oldest trick in the book.

Let me put my conclusion on the table before I make the case for it: I think the people building the most powerful technology in human history are, for the most part, sincere in their fear of it. And I think that fear — whatever its sincerity — has become structurally indistinguishable from a bid for control. When the companies furthest ahead in a race start pleading to have the race slowed down, on terms they help write, the correct response is not to swoon at their conscience. It’s to ask the oldest question in the political economy: Who benefits?

A resignation letter that went off like a flare

A few weeks ago a 27-year-old AI researcher named Jacob Coxon resigned, and did it loudly. He’d spent about three years building frontier models and the last four months of them at Anthropic, the firm he walked out of whilst posting a thread accusing both Anthropic and his former employer OpenAI of “gambling with our lives.” They are, he said, “racing straight to self-improving super-intelligence,” and on current trajectory things could be “out of control” by the end of next year. The post was viewed more than seventy million times within a few days.

Now, the cynical read writes itself: disgruntled kid quits, grabs his fifteen minutes. He told the Wall Street Journal that Anthropic’s safety work is, in his own word, “sincere.” This wasn’t a grift, and that matters more than it first appears, because it frames the entire puzzle of this issue: sincerity and capture are not opposites. A person can be genuinely terrified, and the structure around him can still be quietly pouring a moat.

Anthropic’s own alignment lead has publicly put the odds of AI killing all of humanity above ten per cent. A person whose literal job is to keep the thing safe, telling you there’s a better-than-one-in-ten chance it kills everyone, then going to work the next morning to build it anyway.

Now, I want to take this seriously, and then I want to take it apart. Both in that order.

The part I won’t wave away

Here’s where I break from the crowd I’m about to join. The lazy version of my argument is “it’s all doomer hysteria, ignore it.” That’s wrong, and it’s wrong in a way that will get you laughed out of a serious room.

There are two completely different claims wearing the same coat. The first is the science-fiction one: a superintelligence wakes up, decides it doesn’t need us, and switches us off. That claim is unfalsifiable, endlessly deferred, and conveniently impossible to disprove, which makes it the perfect fuel for a permanent state of alarm. I have very little time for it.

The second claim is concrete and boring and real: that these systems are becoming genuinely good at things we’d rather they weren’t good at, like finding and exploiting software vulnerabilities at scale and running deception. That risk isn’t science fiction. It’s engineering. And notice that even the industry’s loudest sceptic on regulation, Jensen Huang doesn’t deny it, in fact, his entire argument is that the labs must build hard controls against exactly these failure modes. Nobody serious is saying the beast is imaginary. The fight is over who holds the leash, and who gets to decide.

Follow the incentives

This is a behavioural economics newsletter, so let’s do what we always do: ignore what people say and look at what the structure rewards.

In September, Anthropic’s CEO Dario Amodei published an essay titled “We Must Pace the Frontier,” an argument that the whole industry should deliberately slow the rate at which it makes AI more capable. It went off like a bomb: thirty-six million views in a day, and near-instant endorsements from Sam Altman, Elon Musk, Satya Nadella and Demis Hassabis. It’s the third in a sequence; back in January he’d published a twenty-thousand-word warning that AI was about to “test who we are as a species,” and by June he was calling for binding, enforceable, aviation-style regulation of frontier models. The concrete commitment this time was to give outside evaluators permanent, insider-level access to Anthropic’s systems.

Independent testing? I’m genuinely for that. We’ll come back to it. But look at the shape of the broader ask to slow the frontier, regulate the frontier, and then look at who’s making it. It is being made, almost exclusively, by the four or five companies who are already at the frontier, several of them circling historic public listings. When the market leaders in any industry start lobbying for rules about how hard everyone is allowed to compete, an economist doesn’t reach for admiration. They reach for the phrase regulatory capture; the well-worn move where incumbents use the language of public safety to build a moat that only they can afford to cross.

And you don’t have to take my word for the motive. Michael Burry called the pacing warnings “self-serving hype tied to IPOs.” Jack Dorsey published a direct rebuttal arguing for open weights and reproducible testing rather than industry-wide limits negotiated by today’s incumbents. Even Stability AI’s founder called Dario’s specific plan structurally hollow. And Jensen Huang, who builds the chips every one of these labs depends on and therefore has no earthly reason to talk his own customers down, said the apocalyptic claims from some lab leaders simply make no sense to him, and may be driven by other motives. The critique isn’t fringe. It’s coming from inside the tent.

Here’s the mechanism, stated plainly. If the US ends up with a regime where the big domestic labs are regulated but permitted, while open-source models and smaller challengers get strangled by compliance costs they can’t carry, you will not have made anyone safer. You’ll have taken a field with dozens of competing players and concentrated it into a cartel of three. A cartel now operating with the moral licence of “safety” and the political cover of Washington. Concentration of power is the thing the doom narrative claims to fear most. And the proposed cure delivers exactly that.

We talked in the Empire Trap issue about how the smartest incumbents stop competing on the product and start competing on the rules. This is that, at civilisational scale.

The genie doesn’t go back in

There’s a second problem with “let’s all agree to slow down,” and it’s more basic: there is no we.

China has serious open-source models such as, DeepSeek, Qwen, GLM and others released to the world with the weights open, downloadable and un-recallable. That is not a genie you put back in the bottle by having four American CEOs sign a pledge. Nadella made a version of this point on the All-In podcast recently, arguing that China should care about the same safety concerns the US does: “Why should it be different for them?” It’s a fair hope but it is not a plan. A safety framework that only binds the people who agree to be bound, while the frontier keeps advancing in the open everywhere else, doesn’t reduce the risk. It just relocates the lead.

Which is the quiet absurdity at the centre of the whole debate. The single loudest, most anguished conversation about pumping the brakes on AI is happening in precisely the one country whose companies would be handicapped by pumping them, al the while the models proliferate freely elsewhere. You can believe the risk is real and recognise that unilateral American restraint is a strategy for losing the lead without making the world any safer. Both things are true at once.

The cost of the brake

There’s a moral sleight of hand in the word “pace,” and I want to drag it into the light, because it does a lot of quiet work. “Slow down” sounds cautious, responsible, grown-up. But a slowdown is not free and it is not abstract. It has a body count of its own.

Consider what this technology is already doing while we argue about whether to throttle it. At the Cleveland Clinic, an AI model deployed to catch sepsis, which is the leading cause of death in hospitals, and a killer precisely because it disguises itself as almost everything else has cut the mortality rate by 41 per cent. Rolled out across American healthcare, a model like that could save on the order of 140,000 lives a year. AI is designing antibiotics against drug-resistant bacteria and pulling the detection of pancreatic cancer forward by up to three years, a disease that kills 52,000 Americans annually precisely because it’s almost always caught too late. This is not Silicon Valley brochure copy. It is happening now.

Here’s the part the pacing crowd rarely reckons with: you cannot surgically slow only the dangerous capabilities. Progress is entangled. The same advances that make a model better at spotting a tumour make it better at everything, and a blanket instruction to “pace the frontier” throttles the cures alongside the risks.

The better answer already exists

So, let me put a constructive model on the table, and it’s a sharper one than the labs are offering themselves.

Start with the principle the sensible people across every camp actually share: safety is an engineering problem before it is a legal one. The risk concentrates in the frontier labs because that’s where the compute and the capability sit, so the responsibility sits there too. The labs own the safety of their product the way a car maker owns the safety of its cars, through hard engineering discipline: sandboxes, runtime monitors, continuous verification. Even Mark Zuckerberg is right that the labs already carry powerful natural incentives to get this correct: liability, market forces, and user trust, which increasingly is the product.

The cleanest articulation I’ve seen came from Naval Ravikant, in a single line:

“The best way to pace the frontier is to hold the labs fully liable for the behavior of their models.”

That’s the whole mechanism. Not a slowdown negotiated in a boardroom but liability. Make the people who build and profit from these systems legally answerable for what the systems do, and you align the incentive without appointing a gatekeeper. Liability scales naturally to open source and to new entrants. It requires no club. And it can’t be captured the way a rule-writing committee can, because it’s enforced through courts and markets rather than by the incumbents themselves.

Which is exactly why I’m wary of the fashionable alternative: the proposal, floated by Musk and others, that the leading labs test one another’s models before release. It sounds rigorous but peer review among the four companies at the frontier is, on inspection, those four companies forming a club with the power to bless or block each other’s releases. That isn’t a check on the cartel. That is the cartel, in a lab coat. Market-based accountability, full liability, plus genuinely open models keeping the closed ones honest will do the same job without handing the current winners a permanent veto over everyone who comes next.

The serious objection to all this runs the other way, and I’ll give it its due. Scott Galloway argues that self-regulation of any flavour is “performative bullshit,” and that what we need is public authority with real teeth; an international agency modelled on the one that governs nuclear weapons, on the logic that the Cold War proved even sworn enemies will cooperate on a technology that can kill everyone. It’s the strongest version of the case against me, and it isn’t stupid. But the analogy snaps at the one joint that bears the weight. Nuclear weapons are governable because the physical inputs are scarce, you cannot enrich uranium in a garage, and the entire non-proliferation regime hangs on controlling a few tonnes of fissile material across the planet. A model’s weights are a file. They copy infinitely, at zero cost, and China’s best are already open and downloadable tonight. You cannot build an atomic-energy agency for something that has already escaped into everyone’s hands. The nuclear regime worked because the genie could be kept scarce. This genie can’t be, which is why liability, not licensing, is the tool that actually fits the thing we’re holding.

The bet underneath it all

Strip away the noise and this is a decision made under radical uncertainty, which, as we covered a couple of issues back, is the only kind of decision that actually tests you. Nobody knows the true probability of catastrophe. Nobody. Both camps are placing bets dressed up as certainties, and both are quietly resulting: pointing at whatever hasn’t-blown-up-yet as proof they were right all along.

The honest position is the one Geoffrey Hinton, the closest thing this field has to a founding father, who walked away from Google to warn about exactly this landed on:

“Anybody who says it’s all going to be fine is crazy, and anybody who says the machines will inevitably take over is crazy too. Hold both at once.”

The risk is real enough to demand serious, verifiable engineering discipline from the people building it. And the proposed cure of a slowdown authored by incumbents, enforced on some and not others, is very likely worse than the disease, because it trades a diffuse, manageable danger for a concentrated, entrenched one.

A brief word from home, because it’s never far from my mind. Australia has no frontier lab, no seat at this table, no vote on the rules. If this ends in a captured American oligopoly blessed by Washington, we don’t just lack exposure to the upside, we become permanent rule-takers inside someone else’s cartel, governed by safety standards written by companies we don’t own in a country that isn’t ours. Whatever shakes out, it will be decided for us, not by us. That’s a thin place to be standing, and worth naming even if the fix is beyond a single newsletter.

Parting Shot

So here’s the question I’d leave you with: when someone who is winning a race asks you to slow it down for everyone’s safety, how confident are you that you can tell the difference between their conscience and their interest, and what would you need to see to know for sure?