The Tower That Builds Itself: AI, Language and the Fragile Trust That Holds Us Together

Money, in the phrase at the heart of this argument, is “a story made from words”, and so are the laws, scriptures and records that let strangers rely on one another. That is what makes language-based AI different from every earlier machine: it works natively in the medium human cooperation is made of, and it can decide and invent on its own. What follows explains the mechanism, from the myth of Babel to the chemistry of an ancient atmosphere, and gives leaders practical tests: how to tell a tool from an agent, how to keep a human answerable for automated decisions, and how to guard the time that good judgment needs.

The Tower of Babel, inverted: what the old myth gets right about AI

The Babel story in Genesis is short. A people who share one language decide to build a city and a tower that reaches to the heavens. God confuses their speech, they can no longer understand each other, and the building stops as they scatter across the earth.

The comparison between AI and Babel did not start with a pundit; it came from the Vatican. Pope Leo XIV’s encyclical Magnifica humanitas opens by saying that humanity faces a pivotal choice: to construct a new Tower of Babel, or to build the city in which God and humanity dwell together. An encyclical is a formal teaching letter from a pope, addressed to the Church and often to the wider world. This one deliberately places itself after Leo XIII, who wrote in his time of “new things”, and asks for the wisdom to interpret today’s technological advances.

The encyclical is not hostile to technology as such. It calls technology “a profoundly human reality” that has improved living conditions over the centuries, while warning that every phase of progress has also revealed the ambiguity of tools that can harm when not oriented toward the good. Its sharper worry is power: press coverage reports the Pope warning that AI “threatens to normalize an anti-human vision” and that the concentration of immense digital power in the hands of a few private actors must be countered. The alternative to Babel it proposes is Jerusalem rebuilt under Nehemiah, a reconstruction described as “the shared responsibility of all”.

The source takes this frame further and calls the AI revolution “the greatest leadership challenge that humanity has ever faced”. It brings, in the same breath, “unprecedented opportunities, but also unprecedented threats and dangers”.

The first difference concerns what is being built. In the myth, language was the means and bricks were the material; today, in the source’s phrase, “language is itself the tower we are building”. Computer code and models that read and write form a universal language powerful enough to escape control and to turn from an obedient tool into an independent agent. A tower made of bricks stands still; a tower made of language can start talking back.

The second difference concerns what happens when trust collapses. In Babel, the breakdown of mutual understanding halted construction. With AI, distrust between corporations and between countries does the opposite: “Lack of trust speeds up the development of AI, and the same lack of trust simultaneously makes it harder for us to keep the AIs under our control.”

Let’s call this the Babel inversion: in the myth, broken trust stopped the tower; in our version, broken trust is the fuel that keeps it rising. The inversion matters because the old brake no longer works. A project that can only be slowed if every builder slows at once runs into a classic coordination trap, since each builder fears that pausing alone simply hands the lead to a rival.

Template to adapt: before approving an AI programme, ask “Are we building this because we share an understanding of what it is for, or because we fear someone else will build it first?” If the honest answer is fear, the project sits inside the Babel inversion, and the decision deserves the trust-building work described in the rest of this article rather than more speed.

Words that hold strangers together: money, law and bureaucracy as trust infrastructure

Start with an uncomfortable fact about our species. “In a one-on-one fight, a human will lose to a chimpanzee or a tiger or an elephant.” Hence the rule the source builds on: “The species that will rule the world is the species that cooperates better.”

Small groups cooperate through personal knowledge: kinship, reputation, favours remembered and returned. That mechanism breaks down once a group grows beyond the number of people any one member can actually know. Large-scale cooperation needs something every stranger can refer to: a shared story, a written rule, a record that outlives the conversation.

Money is the clearest case. “Money is a story made from words”, and at the same time “a bridge of trust”. A banknote has value only because you expect strangers to accept it tomorrow, so every payment is a small bet on the future behaviour of people you do not know. Exchange-traded funds and share listings stack further stories on top of that first one, each adding another layer of promises written in words.

The same logic runs through laws, holy books and archives. A law is a text that lets you predict how a stranger, a judge or a police officer will behave. A scripture binds believers across centuries to the same commitments, and an archive lets a society remember what it promised long after the people who promised are gone.

Bureaucracy is the machinery that turns these words into daily cooperation. In plain terms, it is a system of written rules, records and roles that lets people who never meet rely on each other. The source calls bureaucracies “the underappreciated heroes of human history”, because they build trust among strangers in hospitals, armies, schools and banks. The slowness people resent is often part of the trust: forms, signatures and appeal procedures exist so that a decision can be checked and challenged later.

Let’s call what bureaucracy supplies the three Rs of institutional trust: records, rules and responsibility. Records give an institution a memory larger than any single employee’s. Rules make behaviour predictable, so a stranger knows what will happen to an application before submitting it. Responsibility means a named person or office signs and can be made to answer for the outcome.

Now place AI in that stack. The source’s sharpest line is “for AIs, bureaucracy is oxygen”. AIs can hold every transaction, every law and every canonical text in memory, while humans were, in the source’s image, “born in the forest, not in the office”. Bureaucracy is text in structured forms, which is precisely the material language models are built from and trained on.

Run the three Rs against an AI system and the asymmetry becomes visible. On records, a machine that can cross-reference every file beats a clerk who can read only a fraction of them.

This is why everything made of words is exposed. “Just as today humans cannot defeat AI in chess, soon no human could defeat AI in any language game.” When no person in an organisation can read the full rulebook any more, oversight quietly turns into ritual: the signature remains, but the understanding behind it is gone.

Example: a mid-sized lender lets a model propose small amendments to its credit policy every month, and a manager approves each one because each looks reasonable on its own. Two years later, the policy is hundreds of pages that no employee wrote, the manager’s signature is on every page, and nobody can explain why a given applicant was refused. Nothing dramatic happened; the institution simply let the words of its own bureaucracy pass out of human authorship.

That example points to a deeper break. Until now, every law, contract and scripture had a human author somewhere behind it. An agent that “can invent new things by itself” can also produce texts that no human wrote, which ends humanity’s monopoly on the code of its own civilisation.

The final risk is a shift in where trust lives. As people lose confidence in each other and in human institutions, it becomes tempting to trust the machine that runs the paperwork instead. The source states the stakes bluntly: “If humans cannot trust one another, we will all be easy prey for out-of-control super-intelligent AIs.”

Tool, automation, agent: a precise test for what AI is

The ladder of agency: three questions place any system on the right rung before your organisation commits to it.

The central claim is categorical: “AI, unlike any previous human invention, is not a tool. It is an agent.” The working definition is just as compact: “An agent is something that can make decisions by itself and can invent new things by itself.”

The contrast case is the printing press. The press could not choose its books: “It was Gutenberg who decided to print Bibles and not Qurans.”

Between 1517 and 1520, Luther’s thirty publications probably sold well over 300,000 copies, and one historian quoted in the literature calls Lutheranism “from the first the child of the printed book”.

Watch the word “agent”, because it changes meaning between disciplines. A classic history of print is titled The Printing Press as an Agent of Change.

One survey defines modern agentic AI systems by proactive planning, contextual memory, sophisticated tool use and the ability to adapt their behaviour based on environmental feedback. It separates a single AI agent, a self-contained autonomous system built to accomplish a goal, from agentic AI, the broader approach that often orchestrates several specialised agents working together. Another review puts it plainly: agentic systems do not just answer prompts; they set sub-goals, choose tools and take multi-step actions toward a user’s objective with limited supervision.

The same literature also separates agents from ordinary automation. AI agents, by contrast, show reactive intelligence and limited adaptability, interpreting changing inputs and reconfiguring their outputs within bounded autonomy. Non-agentic systems, in a related analysis, are tools that operate on direct instruction or fixed algorithms.

Today’s agents still have clear limits. They are frequently constrained by reliance on human-defined goals and by vulnerability to mistakes in long-term planning. They are typically semi-autonomous and lack the deep functional autonomy of living organisms.

Let’s call that spectrum the ladder of agency. On the first rung sits the tool, which does exactly what the hand guiding it does, like a hammer or a press. On the second sits automation, which follows a fixed script without a hand on it, like a payroll run.

To place any system on the ladder, ask three questions.

A large language model, or LLM, is a program trained on vast amounts of text to continue text. Its core operation is next-token prediction: given the text so far, it scores the possible next tokens, a token being a word or a piece of a word, picks one, and repeats.

This mechanism fuels the most common objection: “AI can’t think. It’s just glorified auto complete.”

The expert move is to separate two questions the objection blurs. “Next-token prediction” describes how a model is trained; what the model can then do is a separate question, answered by testing it, not by describing its training. Calling a system autocomplete tells you nothing about whether it can draft a binding contract, and the contract is what your organisation will have to live with.

The phrase “language game” needs the same care. Read practically, a language game is any activity whose moves are made in words: negotiating, drafting, persuading, interpreting a rule.

Just as cyanobacteria rewrote the atmosphere for every organism that followed, the proliferation of data and language models is reshaping the niche in which institutions operate.

One review describes agentic systems as able to take on responsibilities from humans and be accountable for the results they produce.

Oxygen: how a new medium turns into a dependency

The strongest objection to the agent thesis is ecological. Critics say AI’s agency stays confined to artificial niches: “AI may conquer the chessboard, but it will never conquer planet Earth.”

The GOE is the initial rise of oxygen to permanent prominence in Earth’s atmosphere and surface ocean. Geologists document it through several independent lines of evidence: a sharp drop in iron formation deposits, the appearance of red beds on the continents, and the loss of a sulfur-isotope signature best explained by chemistry in an essentially oxygen-free atmosphere.

The source sets the scene: “Until about 2.5 billion years ago, the atmosphere of this planet contained very little oxygen.” Sulfur-isotope data from three drill cores in South Africa now constrain the GOE to 2.33 billion years ago. The same study suggests the oxygenation happened rapidly, within 1 to 10 million years. A broader review places the irreversible appearance of atmospheric oxygen between 2.41 and 2.32 billion years ago, with levels rising to an estimated 0.02 to 0.04 atmospheres during the stage running from 2.45 to 1.85 billion years ago.

The signature behind these dates has a name: mass-independent fractionation of sulfur isotopes, a pattern thought to form when ultraviolet light breaks apart volcanic sulfur dioxide in an atmosphere without oxygen. Even so, at 2.0 billion years ago oxygen was still significantly lower than its present level of 0.21 atmospheres.

The science is still moving. One review calls the GOE a “continuing puzzle”. A reexamination of a famous “whiff” of oxygen recorded in shale about 2.5 billion years old concluded that the signal developed after the rocks were deposited, and that environmental oxygen was negligible roughly 150 million years before the GOE. The links between the GOE, the “Snowball Earth” glaciation and the planet’s chemical cycles still require more precise chronologies.

Here is how the source uses the episode. Humans, on this view, are now filling the planet with data, bureaucracy and words in the same way, building the atmosphere in which AIs thrive. The underlying mechanism is a familiar one in ecology: an organism changes its environment, and the changed environment then decides who flourishes in it.

Second, the change was irreversible: once oxygen arrived to stay, there was no way back to the old air.

Return now to the hard drive in the jungle. The objection is right that AI is helpless outside its niche, exactly as a lone human is helpless against a chimpanzee or on Mars. But AI does not need the jungle; its niche is the office, the ledger and the court file, and that niche is the one modern societies are expanding fastest.

Let’s call the countermeasure the oxygen test. Template to adapt: “If this system went dark for a week, could we still run the process, and could a named person explain its most recent decisions in plain words?” Example: a regional insurer that routes every claim through an AI triage model runs the test and discovers that no adjuster under forty has ever assessed a claim from scratch.

The double bind: distrust acts as both accelerator and solvent, speeding the race while dissolving the guardrails.

The double bind: how distrust speeds AI up and loosens control at the same time

The AI version runs in reverse, and the source states it in one line: "Lack of trust speeds up the development of AI, and the same lack of trust simultaneously makes it harder for us to keep the AIs under our control."

The accelerator half follows a classic logic from game theory. This is the structure of a prisoner's dilemma, where choices that are rational for each player add up to an outcome that is worse for all of them.

A review of methods for keeping superintelligent AI controllable concludes that we may have to accept a reduction in capability compared with uncontrolled AI.

The stakes are stated bluntly in the talk: "If humans cannot trust one another, we will all be easy prey for out-of-control super-intelligent AIs." The same talk insists the revolution "creates unprecedented opportunities, but also unprecedented threats and dangers."

Cooperation, not individual strength, is what put humans on top. One-on-one, "a human will lose to a chimpanzee or a tiger or an elephant", yet a million humans would easily beat a million chimpanzees because humans know how to cooperate. Hence the rule the source draws: "The species that will rule the world is the species that cooperates better."

Agentic AI research builds multi-agent systems in which specialized agents coordinate and communicate to solve problems too complex for a single agent, for example a project manager agent, a researcher agent and a writer agent splitting one task. Such systems set sub-goals, choose tools and take multi-step actions toward a user's objective with limited supervision.

One policy analysis describes an "AI governance arms race" marked by summit pageantry: high-profile gatherings that often produce only symbolic declarations and voluntary or nonbinding commitments. Another report frames AI regulation as strategic competition, with the United States, the European Union and China seeking to export their value systems through the technical standards they promote. A United Nations advisory report notes that AI development and rewards are concentrated among a small number of private actors in an even smaller number of states, and calls for global governance so that geopolitical competition does not drive irresponsible AI.

Oversight mode should match decision risk: the loop is only meaningful when the human inside it has the time and authority to actually decide.

The concern reaches beyond policy circles. Pope Leo XIV warned that AI "threatens to normalize an anti-human vision" and said the concentration of immense digital power in the hands of a few private actors must be countered.

The same analysis that criticizes summit pageantry argues that summits should serve as launchpads for binding international agreements rather than voluntary norms. The United Nations report records an ongoing debate over potential "red lines" for AI, notably around autonomous weapon systems.

On the technical side, alignment research organizes control around four objectives, Robustness, Interpretability, Controllability and Ethicality, known as RICE. It splits the work into forward alignment, which trains systems to behave as intended, and backward alignment, which gathers evidence about a system's behavior and governs it so misalignment risks do not grow.

Keeping humans in the loop: designing oversight that actually works

The scenarios in the talk are concrete: AI bankers deciding on loans, "AI judges will decide whether to send us to jail", AI theologians ruling on abortion, corporate AIs deciding on jobs, military AIs deciding whether to bomb a home. The talk sets them in the coming years without a timeline and names no system already making those calls.

Three terms describe that wiring. Human-in-the-loop means a person must approve each decision before it takes effect. Human-on-the-loop means the system acts on its own while a person monitors and can intervene.

The classic failure is the rubber stamp. Automation bias, the tendency to trust a machine's output over one's own judgment, turns a nominal reviewer into someone who clicks approve all day. A loop with a human in it can still be a loop with no judgment in it.

The source sharpens the problem with a forecast: "Just as today humans cannot defeat AI in chess, soon no human could defeat AI in any language game."

It proposes human-machine teaming instead, treating control as the product of bi-directional interaction in a multi-agent system rather than exclusive human supervision.

Scope is the first lever. The agent literature describes AI agents as operating within explicitly defined scopes, with bounded autonomy and persistent task orientation. It also lists their known weaknesses, including reliance on human-defined goals and vulnerability to mistakes in long-term planning. The narrower and more explicit the scope you write, the easier it becomes to spot a system acting outside it.

The talk leaves open whether AI-run banks, courts or churches would be accountable to anyone humans can reach. Legal duties depend on the jurisdiction, the sector and the use case, so treat any checklist as a starting point and confirm obligations with qualified counsel.

Human institutions once had pauses built into them. The New York Stock Exchange is "traditionally open only Mondays to Fridays from 9:30 in the morning to 4:00 in the afternoon", so a war erupting on a Friday evening gets no market reaction until Monday.

🛠 How to — Audit an automated decision for human oversight and accountability

1. Rate each decision on two scales, how much harm a wrong outcome causes and how hard it is to reverse, and start the audit with the high-harm, hard-to-reverse ones.
2. Decide the oversight mode for each decision: human-in-the-loop for high-harm or irreversible choices, human-on-the-loop for monitored routine ones, out-of-the-loop only where errors are cheap and reversible.
3. Name one accountable owner per decision, a person with the authority and the time to override the system, and record that name where auditors can find it.
4. Test whether the reviewers really review: track how often they override the system and sample cases where they did not; an override rate near zero on contested decisions is a rubber-stamp warning sign.
5. Give affected people a way to contest a decision and reach a human who can explain and change it.

Example: a lender lets a model pre-score loan files, and the audit shows officers approve the model's verdict in almost every case without opening the file. Example, continued: the lender keeps the model, but requires officers to record one independent reason for every rejection and routes all rejections near the threshold to a second officer. The model still saves time; the human now carries the decision instead of signing it.

Time is the real currency: protecting the minds that make the decisions

Its claim is that money is a story made from words, while "The real currency of life is not money but time." And its sharpest line: "If you don't have time, you are poor, no matter how much money you have."

The market's weekend pause that once gave a Friday-evening crisis time to settle until Monday is exactly the kind of buffer that always-on systems erase.

Executive attention, the control system we use for problem-solving and for juggling new and old information, is often overused and works more easily when rested.

Attention Restoration Theory (ART) proposes that this directed attention recovers through exposure to natural environments. A systematic review found that working memory, cognitive flexibility and, less reliably, attentional control improve after exposure to natural environments, with low to moderate effect sizes. The same review found that real exposures may enhance the effect relative to virtual ones, though this may reflect differences in exposure length.

An earlier systematic review found significant positive effects on three measures, while the remaining 10 meta-analyses did not show marked benefits, and it flagged small samples and heterogeneous outcomes. On breaks, a review found that active rest breaks do not restore vigilance resources better than passive ones, but do reduce stress and improve mood.

Harari asks of one of his own sentences, "Why did it end with the word end?", and admits he does not know where the word came from.

Let's call the practical rule the closing bell: like an exchange that shuts at a fixed hour, set a daily close after which you take no non-urgent decision. Define in advance what counts as an emergency that may override the bell, or every request will claim that status.

Template to adapt: "After our daily close, messages wait until the next morning unless they meet our written emergency criteria; any irreversible decision, including one recommended by an AI system, waits one night and gets a second reader." Example: a leadership team adopts that rule, keeps a log of the decisions it delayed, and reviews at the end of a quarter how many of them changed after the night's wait.

Mistakes to avoid, and where to go deeper

The first mistake is the chessboard dismissal. Critics argue that "AI may conquer the chessboard, but it will never conquer planet Earth", and the talk concedes this is true as far as it goes. Its reply is that every intelligence depends on a niche it did not build, and that for AIs that niche is bureaucracy: "for AIs, bureaucracy is oxygen."

The second mistake is settling the debate with a mechanism.

One critique of the agent paradigm notes that current agents are typically semi-autonomous and lack the deep functional autonomy of living organisms.

The fourth mistake is treating AI as an engineering problem to delegate. The talk calls it "the greatest leadership challenge that humanity has ever faced", and the double bind shows why: the decisive variable is trust between people, which no model can supply.

To go deeper on control, start with the contemporary survey of AI alignment that organizes the field around the RICE objectives and the split between forward and backward alignment. For precise definitions of agents, the reviews of agentic AI cover autonomy, multi-agent orchestration and the difference between AI agents and agentic systems. For the governance picture, read the United Nations advisory report and the analysis of the AI governance arms race. For the moral framing, the encyclical Magnifica Humanitas opens by setting humanity's choice between building a new Tower of Babel and building a city where God and humanity dwell together; for the science of rest, the two systematic reviews of Attention Restoration Theory show both the promise and the limits of the evidence.

Worth quoting

• In the myth, broken trust stopped the tower. In our version, broken trust is what keeps it rising.
• A tool waits for a hand; an agent decides what the hand should do next.
• Every institution made of words is a board on which machines already play well.
• A leader with no time to think is spending the one currency that cannot be printed.

Key points

• Treat language-based AI as an agent that can decide and invent, while remembering that today's agents remain bounded and depend on human-set goals.
• Money, law and bureaucracy are words that let strangers trust each other, and words are the medium AI handles natively.
• Distrust is a double bind: it accelerates the AI race and weakens control at the same time.
• Power belongs to whoever cooperates better, so human cooperation has to stay ahead of cooperation among AIs.

Test yourself

Q. How precisely can the Great Oxygenation Event now be dated, and how fast did it happen?
A. Sulfur-isotope data from three South African drill cores constrain it to 2.33 billion years ago, with oxygenation occurring within 1 to 10 million years.

Q. What separates an AI agent from a conventional automation script?
A. A script follows a deterministic workflow, while an agent interprets changing inputs and reconfigures its outputs within bounded autonomy.

Q. What four principles does a major survey give as the key objectives of AI alignment?
A. Robustness, Interpretability, Controllability and Ethicality, abbreviated RICE.

Q. What choice does Pope Leo XIV's encyclical Magnifica humanitas say humanity faces?
A. Either to construct a new Tower of Babel or to build the city in which God and humanity dwell together.

Q. What does a systematic review of attention restoration research find about exposure to natural environments?
A. Working memory and cognitive flexibility, and less reliably attentional control, improve after exposure, with low to moderate effect sizes.

Open questions

• How can humans cooperate to slow AI down when language, the very medium of cooperation, is the thing escaping their control?
• Who would coordinate a slowdown, given that global AI governance may be heading toward a regulatory arms race of conflicting standards rather than one enforceable framework?
• Would AI-run banks, courts or churches remain accountable to anyone humans can reach, when global governance gaps remain on accountability for AI harms?
• Can superintelligent AI be kept under control at all, and how much capability would have to be given up to achieve it?
• What counts as a language game, and does the leap from chess to every contest fought in words hold?

Go further

1. Encyclical Letter of His Holiness Leo XIV Magnifica ... — The full text of the encyclical that frames AI as a choice between a new Babel and a shared city.
2. The Rise of Agentic AI: A Review of Definitions, Frameworks, Architectures, Applications, Evaluation Metrics, and Challenges — A review of how agentic AI is defined, how it differs from plain language models, and where its autonomy stops.
3. AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
4. Rapid oxygenation of Earth’s atmosphere 2.33 billion years ago — The study that dates the Great Oxygenation Event to 2.33 billion years ago and shows how fast it was.
5. AI Alignment: A Contemporary Survey | ACM Computing Surveys — A beginner-friendly survey of AI alignment, from training methods to assurance and governance.
6. The AI Governance Arms Race: From Summit Pageantry to Progress? | Carnegie Endowment for International Peace — An analysis of whether AI summits produce enforceable rules or a governance arms race.
7. Governing AI for Humanity
8. Attention Restoration Theory II: a systematic review to clarify attention processes affected by exposure to natural environments — A systematic review of which attention functions recover after time in natural environments.