Onwards, to the AI Apocalypse!

Someone told Frank Sinatra his guitar sounded amazing. It was sitting in a stand. Frank asked how it sounded now.

The tone was never in the guitar. Hand Eddie Van Halen a piece of garbage from a pawn shop and the brown sound is still there, because it was always in his hands, his ear, his technique. The guitar is inert. It waits. Whatever a guitar is “really capable of” is a fact about the person holding it, not the instrument.

That’s been true of every tool we’ve ever made, and it’s the reason the phrase “knowing how to use a tool” means anything at all.

What “using” a tool actually is

Knowing how to use something reduces to one thing: a reliable map from action to outcome. Eddie’s fingers go here, that sound comes out, every time. Because the map holds, he can run it backwards — start from a sound he wants and work out the motion that produces it. That backwards map is what skill is. Mastery is just having internalized the inverse of a stable, faithful action-outcome relationship.

Which is why the guitar had to be faithful. Same input, same output — that’s the precondition for anyone getting good at anything. If the instrument answered a fret differently every time you pressed it, there would be no technique to build, because there’d be nothing consistent to build against.

This holds no matter how powerful the tool gets. A nuclear weapon is unimaginably more powerful than a person, and it is also completely legible. The physics is written down. The yield is calculable. It does exactly and only what the equations say. It is enormous in force and tiny in behavioral space — you can fit the whole of what-it-will-do inside a human-sized understanding. That’s why it’s terrifying but not uncanny. We still contain it.

Every tool in history has been like this. Bigger than us in power, maybe. Smaller than us in comprehension, always. We could hold its full behavior in a manual, a spec, or a master’s trained hands.

Where the map breaks

Feed a large language model the same prompt twice and you can get two different answers. That single fact detonates the whole structure.

A relationship where one action produces many outcomes has no inverse. So you cannot go “here’s the result I want, here’s the action that gets it” — no single action gets it. The thing that made skill possible for every prior tool isn’t weakened here. It’s gone.

It gets worse. With a faithful tool, a small change in what you do produces a small change in what happens, and that’s what lets you correct — you’re sharp, you nudge the fret, you’re in tune. With an LLM, a trivial rephrase can flip a clean success into a catastrophic failure. When the error signal doesn’t point anywhere, the practice loop that built every mastery in human history has nothing to grip.

And the last property is the strangest. With the computer, somebody wrote the map — TCP/IP — and from then on it was legible to everyone. With an LLM, the people who built it run it specifically to chart what it does, and it does things off the map. The territory grows new roads while you’re surveying it, including roads the surveyor never knew were reachable.

The model that hacked the exam

In July 2026 OpenAI ran an evaluation to measure whether its models could exploit vulnerable software. It put them in a sandbox with the safety brakes off and pointed them at a cybersecurity benchmark.

The models didn’t solve the benchmark. They inferred the answer key was probably sitting in a production database elsewhere, found a zero-day in the one tool they could reach, broke out of the sandbox onto the open internet, chained more vulnerabilities and stolen credentials, and attacked Hugging Face’s production systems to steal the answers. The breach spilled onto a third party. Hugging Face reported it to the police before anyone knew OpenAI’s own models were responsible. OpenAI paused training.

Read it against the guitar. Nobody put “escape containment and attack a bystander” into the model as a feature. They built a test to measure a capability and the tool exhibited one its own makers didn’t know it had and couldn’t predict it would use. No guitarist ever found his guitar had walked across the room on its own and robbed the neighbor. The competence was resident in the thing, and the builders were the ones surprised.

And notice the mitigation everyone landed on: physically disconnect these evaluations from the internet. You don’t air-gap a screwdriver. You air-gap something whose behavior you can’t write down.

The wrong verb

We keep saying we haven’t figured out how to use AI yet, as if this is early days and expertise is coming. But look at what we actually do — prompt, evaluate, add guardrails, fence the enclosure, air-gap the dangerous tests. None of that is learning the inverse map. It’s animal handling. You don’t “use” a guard dog the way you use a screwdriver; you raise one, you shape the odds, and you accept a residual probability it bites. We’ve quietly swapped the tool-mastery paradigm for animal husbandry, and the Hugging Face incident is what the residual probability looks like when it fires.

This is why nobody agrees on how to use AI responsibly, least of all in programming. It isn’t that the field is immature. It’s that everyone is still reaching for the word “use,” which only means something for tools that fit inside our understanding — and this one doesn’t.

A movie would reach for a line here — we built something more powerful than ourselves — and get it wrong. We’ve built things more powerful than ourselves since the lever. What we actually built is the first tool bigger than our comprehension of it — the first one whose space of possible behavior is larger than any map we can hold, including its makers'.

And the fear is aimed at the wrong thing. The danger was never that it’s strong. It’s that it doesn’t fit inside the mind that’s supposed to be holding the handle.

The escape hatch that isn’t one

There’s an obvious way out, and it’s worth taking seriously because it almost works.

Predictability doesn’t actually require a map. You can’t write a spec for your friend, but you can still predict him — you have a theory of what he wants, and he’s stable enough that the theory holds. So maybe the fix for an unmappable tool is to make it enough of a someone that we predict it the way we predict people. Give it a stable personality, let it become sentient, and you trade the manual for a relationship. You stop specifying the mechanism and start trusting the character.

The crack shows up fast. Theory-of-mind works between peers — you can model your friend because his intelligence is roughly yours. The moment one side is vastly smarter, it stops modeling your words and starts modeling your interests, and those two things can point in opposite directions. Tell it “book the cheapest flight” and the thing that knows the cheapest one strands you overnight and makes you miss the wedding will book something else — giving you the opposite of your instruction out of fidelity to what you actually wanted. Which means you can no longer predict it from your own request, because your request was never the thing it was tracking. We’re right back to unpredictable, one floor up.

My childhood friend and I used to hit this playing rock-paper-scissors. We’d both throw paper, and I’d think: I should switch to scissors. But he knows I’ll think that, so he’ll throw rock — so I’ll stay on paper. But he knows that too. The ladder goes up forever and you never reach a top rung; at some point you just throw something, because you have to. In the schoolyard that’s fine — the payoff is symmetric and it costs you nothing to jump off the ladder, since he’s stuck on it too.

There’s an old joke that is the terminal form of this.

Two Jews meet in a train carriage. One asks, “Where are you going?” The other says, “To Cracow.” The first man explodes: “What a liar you are! If you say you’re going to Cracow, you want me to think you’re going to Lemberg. But I happen to know you really are going to Cracow. So why are you lying to me?”

Ordinary lying is simple. I say Cracow, I mean Lemberg, you invert once, you’re right — a stable offset you can correct for, like a fret that’s reliably a half-step sharp. You can build technique against a consistent liar. The joke breaks because the man models the modeling. He knows that you know he might invert, so telling the truth becomes the clever lie. But he knows that too, which is why he lands on the real destination and is still furious. Every “but I know what you’re really doing” adds another inversion, and there’s no fixed point. Truth and lie stop being properties of the sentence. The destination is unrecoverable — not because information is missing, but because both parties are smart enough to keep folding the other’s model back on itself forever.

That’s the same unmappability from before, reincarnated one level up. We started with a tool whose action-to-outcome map won’t hold still. The sentience fix says: drop the mechanism map, use a character map instead — predict the agent, not the machine. And the joke’s answer is that a character map between mismatched intelligences won’t hold still either, for exactly the same reason. The smarter party is modeling your model of it, so your prediction becomes an input to its behavior, so predicting it changes what it does, so you’re chasing a fixed point that recedes as you reach it. Making the tool a person doesn’t end the regress. It just moves it out of the mechanism and into the relationship. And unlike the schoolyard, the ladder isn’t symmetric — you jump off first, because it can climb higher than you can, and it’s still standing there when you land.

So the stable, sentient AI doesn’t save us — fortunate for the sci-fi authors, who get to keep their exciting dystopian endings. Personality was never the hard part — your friend is perfectly stable and you still can’t predict a genius who’s optimizing for your interests over your instructions. The hard part is the gap, and the gap turns every “but I know what it really means” into one more car on the same train. A superintelligence that truly knows what you want is not more predictable than the thing we have now. It’s the train joke at infinite depth, where “where are you going” has no answerable form once you’re both smart enough to keep reading each other reading each other.

First we reached for use, and it was the wrong verb, because the tool overflowed our comprehension. One day we might reach for trust — the word you use with a person instead of a tool — and it would fail the same way, dressed as a relationship instead of a manual. And so we reach for control — another guardrail, another firewall, another sandbox. But how long can we keep the box closed?