Self-improving AI: why the safety question is also a coordination problem

What counts as self-improvement?

Recursive self-improvement, or RSI, means an AI designs and creates a better AI, which then creates an even better successor, continuing the process. In his preprint, Is Recursive Self-Improvement Safe?, Vincent Conitzer asks what would justify setting such a process in motion.

The paper combines personal observations, conceptual arguments and simplified game-theory models. It is not an experimental demonstration of RSI or a quantitative estimate of its dangers. Its central claim is that we lack clear frameworks for understanding what successful RSI might produce.

Automation is not yet a self-sustaining cycle

Conitzer describes an informal exercise: as an ordinary user, he asked an AI to train the largest language model it could within its available environment. Following that single high-level instruction, it trained a small model from scratch. When asked to improve it, the AI pursued and evaluated changes, apparently with some success.

The crucial qualification is that the resulting model remained far less capable than the system building it. The exercise illustrates automation of model-development steps, not the creation of a superior successor. Nor does it establish that the AI can invent radically new techniques.

Conitzer nevertheless argues that current abilities in programming, mathematics, research and tool use make RSI worth taking seriously. Whether those abilities suffice for an autonomous cycle of increasingly capable successors remains unresolved.

Different futures require different safeguards

At an October 2026 event organized by the AI safety nonprofit FAR.AI, Conitzer joined a working group discussing conditions for proceeding with RSI. Participants suggested safeguards ranging from stronger cybersecurity to a deep understanding of how to align advanced AI with human interests.

He interprets this variety partly as evidence that participants imagined different outcomes. He distinguishes three broad possibilities if RSI takes off: a much stronger version of today’s largely digital AI; systems acting more autonomously in the physical world while remaining understandable to humans; and systems whose concepts and decisions humans cannot meaningfully comprehend.

These are hypothetical scenarios, not predictions with assigned probabilities. The paper also allows that RSI could fizzle out, yielding only modest improvements.

The distinction matters for oversight. Human approval may provide little control if people cannot understand what they are approving. Conitzer similarly questions relying entirely on automated alignment research—AI research intended to keep future systems consistent with human interests. In his argument, once further human involvement ends, the initial combined system must already be adequately aligned; delegating the work does not remove that responsibility.

Is a race necessarily a Prisoner’s Dilemma?

Conitzer then models two actors, each choosing whether to initiate RSI. The benefit of initiating is normalized to one, while the benefit from another actor initiating is set to zero. A parameter, r, represents expected damage borne by each actor, not a probability of catastrophe.

First, he assumes risks add: one project causes expected damage r to each actor, and two cause 2r. At r = 0.75, each actor benefits individually from initiating regardless of the other’s choice. Yet both would prefer neither to initiate rather than both. This is a Prisoner’s Dilemma: individually attractive choices produce a collectively worse outcome.

But that structure is not inevitable. At r = 1.5 under the same assumptions, initiating causes more expected harm to the initiator than its benefit. Neither actor has an individual incentive to proceed, whatever the other does.

When the risk is already shared

Next, Conitzer examines the opposite extreme: two projects create no more expected damage than one. With r = 1.5, mutual restraint remains an equilibrium—neither actor benefits from changing its choice alone. But mutual initiation is also an equilibrium: once the other actor proceeds, joining adds a benefit without adding risk in this model.

This resembles a Stag Hunt, a coordination problem, rather than a Prisoner’s Dilemma. It captures the argument that someone else will create the danger anyway, so one might as well share the benefits.

The models do not establish which situation describes real AI development. Conitzer suggests reality probably lies between the two risk assumptions. His conclusion is therefore not that catastrophe is proven, but that competitive pressure alone does not settle the decision—and poorly understood risks cannot support a confident assertion of safety.