AI race needs a brake pedal

Jacob Coxon, a researcher at Anthropic, recently resigned from the company, citing the industry’s disregard for the risk that artificial intelligence (AI) could annihilate humanity. Days later, Anthropic CEO Dario Amodei published an essay outlining how and why industry and government must come together to pace frontier AI development to avoid it slipping beyond human control. Others quickly agreed.

None of this is new. Geoffrey Hinton, a renowned computer scientist often called the “Godfather of Artificial Intelligence," estimated a 10 to 20 percent chance that advanced AI could end humanity within the next thirty years. While that sounds alarming, it implies an 80 percent to 90 percent chance of survival. So, how do we increase the likelihood of this safer outcome?

There are ways to move humanity toward the safer side of Hinton’s odds. While nothing is guaranteed, a few conditions would make a difference.

Cooperation between the United States and China is an obvious focus, as they are the countries with the greatest influence over AI’s future. Amodei’s proposal for embedded third-party evaluators, capability thresholds and a path to shared standards is a strong baseline, but it cannot come from a single company. Other actors, including governments, need to agree.

The slowdown should be a precondition and immediate goal, buying time to build three mechanisms for evaluation: verification against covert development and deployment, independent assessment of dangerous capabilities and collective incident monitoring with crisis protocols.

There is precedent in China and the United States coming together to avoid shared risks. In November 2024, Washington and Beijing agreed that decisions about nuclear weapons should not be delegated to AI. It did not end the rivalry, nor did it intend to. The focus was a shared concern about nuclear decision-making.

AI security needs a similar channel. Talks between the Treasury in Washington and economic planners in Beijing focus on financial stability and misuse by malign actors. That agenda could expand to loss of control, recursive self-improvement and crisis escalation. With President Xi Jinping expected in Washington this month, and major summits ahead, there is an opening.

These channels should not be improvised. A realistic path runs from voluntary lab-to-lab pledges and action for independent evaluation, possibly reinforced by strong incentives. This should lead to more formal, binding rules and an international enforcement body.

Beyond U.S.-China diplomacy, collaboration among other countries could add balance to the global AI ecosystem. Many “middle powers” want a say in the standards that shape their societies, but lack the computing power for rival frontier models. The United States holds roughly 75 percent of global AI computing capacity, China 15 percent and the European Union about 5 percent.

Pooling resources, an approach some call the CERN for AI, would give them leverage and unlock evaluation and more legitimate standing to participate in verification processes to hold everyone accountable. Instead of trying to build national champions, they could combine public research institutions, computing budgets and technical expertise into shared frontier security programs. They could split the high cost of training models and build systems with embedded safety guarantees.

There are positive early signs. Australia, Canada, the EU, France, Japan, Kenya, South Korea and Singapore already cooperate through a network of AI safety institutes now managed from Britain. Nine of the twelve leading AI companies set thresholds for when a model becomes powerful enough to warrant extra caution, but move the goalposts as the race intensifies for fear of losing their edge. There is no common benchmark — each company decides for itself.

After this summer's AI incidents and last week’s declarations, self-evaluation will not inspire public confidence. Independent experts, resourced to verify that access controls, monitoring and shutdowns work, are the more credible route, much as aviation records near misses to spot danger before disaster strikes.

The final condition is less dramatic, but crucial nevertheless: a shared vocabulary for frontier AI risk to enable coordinated international governance. Though this month’s UN General Assembly will not produce a binding agreement, it should not be judged by that standard. Its value lies elsewhere.

Before governments can agree on what to do, they need to agree on what they are talking about, and then decide how far they are willing to go to prevent catastrophe.

There are precedents in global risk management. Consider Ozone diplomacy. The 1985 Vienna Convention set no binding limits on ozone-depleting chemicals, only common definitions and a forum to compare data. Seven weeks later, researchers published the discovery of a hole in the ozone layer. Two years thereafter, that framework became the binding, enforceable Montreal Protocol.

Definitions and convenings came first. Fast, binding action followed undeniable evidence. It can be the same for AI: voluntary steps and shared definitions first, binding rules once the evidence and the will are undeniable, with regional commitment becoming global.

Much will depend on how Chinese labs and Beijing respond to Amodei's call. Their answer will show whether the willingness to slow down is genuinely shared.

None of this guarantees the safer four-fifths of Hinton’s odds. Scientists still disagree about the scale of catastrophic risk, and governments have abundant reasons to postpone difficult choices. But these steps would, at a minimum, show that we will not resign ourselves to probability.

Hinton’s warning was meant to unsettle us. It should also remind us that most of the future is still undecided, and still ours to shape. But time is running out, and there is no coming back from loss of control.

Nicolas Miailhe is co-founder of AI Safety Connect, a Track 1.5 diplomacy platform convening governments, frontier AI labs and civil society on AI safety and governance.