What should the UK be doing to make AGI go well?

Adam Jones

Personally

This is my personal blog, where I write solely in my personal capacity. It does not represent the positions of any organisations I'm associated with.

Also, everything here is provded "as is", without warranties of any kind, express or implied.

15 August 202515 August 2025

We could end up in an AGI race between the US and China. These race dynamics incentivize cutting corners on safety, and having AGI controlled by a single nation (even a democratic one) concentrates unprecedented power in one place.

The UK and similar states have the opportunity to avoid and mitigate these outcomes - steering AGI to benefit not just their country, but humanity at large.

I’ve broken this down into three things:

• Thing 1: Avoid being in a race in the first place
• Thing 2: If we’re in a race, make it go not bad
• Thing 3: If things go bad, be prepared

(This article assumes you think AGI could pose significant risks, and could come reasonably soon. If not, you probably want to read those first.)

Avoid being in a race

I think the UK can and should reduce the likelihood of a race. Things like:

• Highlight the massive shared risks that come from rushed development. Evals work from UK AISI is valuable here, but exporting/communicating this research is maybe just as important. The UK acting seriously around AGI also could set a tone for others taking things seriously. (The first AI Safety Summit was also good for this, trying to undo the damage done by the Paris AI Action Summit might also be good.).
• Make treaties to discourage dangerous or reckless development (e.g. like the European Bioethics Convention, or Protocol on Blinding Laser Weapons), and improve transparency between states (e.g. similar to the Treaty on Open Skies). The UK, or maybe Singapore, seem especially important for facilitating dialogues between US <> China.
• Encourage non-treaty cooperation. Incentivising the US and China to work together or be dependent on each other could make cooperation more likely. Joint ventures in AI applications and shared infrastructure both countries depend on could make defecting costly. Even just cooperating a bit on some obviously shared goals (e.g. preventing AI-enabled terrorism, or severe global economic shocks from AI) might build the relationship and have positive outcomes. And much of the time the US and China’s goals don't conflict - China focuses primarily on regional influence and domestic stability, while the US has also been shifting to focus on itself, so it’s plausible they can live alongside each other.

As a middle power with diplomatic credibility, resilient institutions, and able to get on with both the US and China, the UK is in an excellent spot to work on the above.

Make a race go not bad

While the above will mitigate some of the worst outcomes, I think a race is still pretty likely.

In a racing world, I worry about the US and China being reckless, and this leading to a bad outcome. I also worry a lot about who controls AGI, and the power concentration risk that comes with this - in particular, if humans are no longer economically relevant, how we stop falling into locked-in totalitarianism (also see my articles on AI-enabled oligarchies, Formation Research, and the Intelligence Curse series).

The UK can't outspend the US or China, but it doesn't need to. By building strategic compute capacity and coordinating with allies, the UK can make itself indispensable at the moment when AGI development becomes a classified, national-security priority.

To do this, the UK and other stable democratic countries (e.g. EU and EFTA members, Canada, Australia, Japan, South Korea) should build significant compute infrastructure able to process top secret classified data. (I think this will become necessary as AGI is seen as a national security project, and model weights, compute multipliers or other IP is classified as top secret).

The pitch to the US would be straightforward: "We can boost your secure compute capacity by 30-40%, giving you an advantage over China. In exchange, we want formal governance rights over how AGI development proceeds - safety requirements, deployment decisions, and benefit distribution."

I’m pretty convinced that having a coalition of stable democracies controlling AGI would be much much better than one or two states (especially if those states are the US and China). Norway managing to avoid the resource curse with its sovereign wealth fund shows having good governance can work!

Other uses for this compute could be:

• Offering compute to China, in exchange for safety guarantees etc.
• Using compute to run AI models, to keep their own economies afloat. This is not as good as enabling global benefit sharing, but it’s likely better to have more countries still having functioning economies for avoiding power concentration. And it also derisks investing now.

Practically, this means countries like the UK should be:

• Unblocking the construction of lots of data centers by private companies. These need to be at bigger numbers than the UK has been celebrating so far - compare OpenAI investing $500Bn over the next 4 years with the UK’s planned £25Bn of investments (Although great directionally! Just more please!).
• Building top secret compute facilities now. This is somewhere the government might have an advantage over the private sector.
• Planning how to convert existing private compute into top-secret compute, or nudging datacenter builders to enable this when planning.
• Trying to loan some of the compute above to major labs to train frontier models (not academics doing very different things at smaller scales!), to understand where they are failing/what the real needs are.
• Encouraging other stable democratic states to do the same, and building relationships with them. An international coalition here could be very valuable.

Be prepared for badness

Things could still go sideways. The UK and similar states could invest in preparing for bad outcomes, and investing in defensive technologies to mitigate their impacts. A lot of this could be done by private companies, given the right environment and encouragement - government should be thinking about how to create that environment.

Defences powered by capable AI agents seem particularly valuable. This is because as AI capabilities scale, the defenses also improve, meaning you’re not caught out when there are sudden jumps. (I imagine most of these to look less like traditional IT systems, and more like virtual AI employees that are really good at e.g. cybersecurity defence, vaccine development. This is NOT ‘cybersecurity using AI classifiers from 10 years ago’, or ‘an LLM reads your logs and flags suspicious stuff’ - it’s ‘an AI agent gives you the equivalent of hiring a team of 10 excellent security professionals’.)

The work AISI is doing around chembio, cyber, human influence, and criminal misuse is useful for understanding the risks. What would be also exciting is adjacent to this: evaluating ‘how good is an AI at cyberoffense’ is close to evaluating ‘how good is an AI at cyberdefence’, which is close to ‘understand how to make AI’s good at cyberdefence’ (especially with reinforcement learning these days). Additionally the societal resilience team’s work adjacent to ‘how does AI proliferate’ is close to ‘how can we encourage the proliferation of defensive AI tools’. The government also has paths in other areas to encourage uptake, for example if the NCSC built AI cyberdefence tools and deployed them as part of their ACD programs.

Conclusion

The UK could steer AGI - if it takes action. Without this, we’re likely to get a dangerous two-player race between the US and China that could threaten billions of lives. Even fairly optimistic outcomes in this scenario aren’t great for the UK.

The UK could develop durable leverage in this narrow window of opportunity by:

• cooling race dynamics,
• building compute and an international coalition around it, and
• preparing for the arrival of AGI.

This could secure the UK and its democratic allies governance rights over AGI development - rights that ensure the safety and security of the UK and its interests, unlock economic growth, and promote fair global distribution of benefits. We’d see AGI development that actually serves humanity - powering breakthrough science, eliminating scarcity, and augmenting rather than replacing humans. AGI could be our next chapter, if we claim our seat at the table.