Two documents appeared a few days apart.

One was Anthropic's When AI builds itself: it argues with data that AI has begun to accelerate its own development — and includes internal metrics not previously disclosed: more than 80% of the code is now written by the models themselves. The other was an encyclical, Leo XIV's Magnifica Humanitas (this piece focuses on its third chapter, which addresses AI), on human dignity in the age of AI.

An engineer and a pope, working in utterly different vocabularies. By rights they should have nothing in common, yet again and again they point at the same subject from different directions. What follows examines a few of those points — and the shared question sitting above them.

1. Recursive self-improvement

While most AI discussion gets stuck on the question “is it smarter than us?”, both texts quietly step past it.

What really changes the game is not a smarter model but a feedback loop — the system becomes a load-bearing part of its own iteration. Anthropic names it outright: recursive self-improvement, and quantifies it: task horizons roughly double every four months, models take on more and more engineering work, and they run experiments that produce the next generation of models.

The encyclical naturally uses no engineering terms, but it invokes one of the oldest images there is: the Tower of Babel. We are used to filing Babel under “hubris” and “height”, yet at its core it is also the story of a project expanding faster than the coordination holding it up — and scattering apart as a result. A loop that escaped its builders. Strip away the theology and what remains is a system diagram:

  • positive feedback

  • no regulating mechanism

  • capability flowing back and reinforcing itself

Anthropic's data is drawing this diagram; the pope is labelling the failure mode.

The author: the threshold that actually matters may not be “superintelligence” but the moment a tool enters recursive self-improvement — a far lower and much nearer threshold.

2. The unknowability of the system

This is common ground that should not be underestimated.

The most technically perceptive sentence in the encyclical is that modern AI is “cultivated rather than built” — developers do not assemble capabilities block by block; they cultivate an architecture, capabilities emerge from the training process, and the system's internal representations remain, in substantial part, invisible even to its developers. This is not a theologian dabbling outside his field; it is an accurate description of frontier AI training.

Anthropic's whole thesis rests on the same fact: these systems are emergent, increasingly autonomous, and already steering part of their own improvement.

The author: so the two agree on the premise — we are deploying systems we cannot fully read — and part ways on the conclusion. For the engineer, opacity and the alignment problem that follows are a problem to be solved. For the pope, they are grounds for humility and for putting on the brakes.

3. Technology is not neutral

The encyclical puts it plainly: technology “bears the face of the people who conceive it, fund it, regulate it, and use it”, and cuts sharper still — “if AI morality is decided by a few, a more moral AI is meaningless.”

This is not vague humanistic criticism. It is a precise statement about how values are embedded in code: what the objective function is, what the reward signal is, what evaluation standard, what data-filtering mechanism. Anthropic's article confirms it without flinching — noting that human engineers still “decide what the problem is and set the grading criteria”. It also describes an automated reviewer enforcing a particular quality standard across the codebase.

The author: the “few” the encyclical warns about are not a rhetorical device but a real, small group setting the standards — a group that includes the authors of the other text.

4. When humanity becomes the bottleneck — where the real disagreement lies

Anthropic invokes Amdahl's Law: speeding up one stage of a process does not remove the constraint, it moves it elsewhere. Their example is concrete: as model code output explodes, human code review becomes the new bottleneck.

The encyclical describes the same phenomenon in ethical language: when “execution” becomes almost free, the scarce resource becomes “decision” — and with it the question of whether anyone still answers for the decision.

So both texts converge on the same node: human in the loop.

But they give that node completely different meanings.

The author: for the engineer it is latency, friction to be continually compressed and optimised; for the pope it is the last resting place of responsibility, a load-bearing wall that must never be knocked out. The disagreement is not about where the bottleneck is, but about whether that bottleneck is a defect of the system or the reason the system exists.

Interestingly, Anthropic's own staff have begun reporting the consequences of this tension.

One remembers the gift economy of small favours between colleagues; another writes that on days when everything ran smoothly, “it didn't matter what I did… but when the system broke, I suddenly realised I no longer knew what I had been doing.”

The author: the people closest to the acceleration are describing, in their own words, the erosion the encyclical fears: as more and more judgement and creation is handed to the system, people lose not only the task but the sense of what the task meant. Slow human judgement, necessary friction, even limited capacity may not simply be inefficiencies waiting to be optimised away — they may be the conditions that make responsibility, participation, and meaning possible.

5. The future is not predetermined

Both texts seem to prophesy disaster, yet the opposite is true.

Anthropic states plainly that recursive self-improvement is not inevitable. The encyclical says technology “is not destiny”. Both reject technological determinism, which means both are making a political and ethical claim: the trajectory is a choice, and that choice is still open.

The author: the two texts are each other's missing half. Anthropic explains how it might happen but lacks a purpose, reducing the answer to “what should we do” to coordination and verification. It looks like governance that does not know what it governs for. The encyclical supplies an explicit teleology: human dignity, the common good. But it mostly treats the technical mechanism as a black box, to be constrained from outside.

Put the two together and we finally arrive at the real question: if intelligence begins to take part in its own creation, what do human beings want it to serve? And who has the right to answer that?

Conclusion

The most fundamental question of the AI era is not whether machines will be smarter than us, nor whether they will replace certain jobs. It is this: as we hand more and more of the building to what we have built, are human beings still the authors of their own future — or are they quietly degrading into regulators of a process they can no longer fully read?

On this point the disagreement between Anthropic and the pope cuts to the core. The former holds that distinctly human judgement, taste, and sense of direction are the next capability frontier to be crossed; the latter holds that they are precisely the bedrock of humanity that technology can never occupy.

So the real choice of our time is no longer technical but civilisational:

Are we building a Tower of Babel that keeps accelerating until it escapes the control of its builders?

Read the originals:

MAGNIFICA HUMANITAS

https://www.vatican.va/content/leo-xiv/fr/encyclicals/documents/20260515-magnifica-humanitas.html

When AI builds itself

https://www.anthropic.com/institute/recursive-self-improvement