A good answer can be dangerous when you do not quite know where it came from. You stop arguing with it. It becomes settled before it has become true.

The same thing happens with code. A neat diff, sensible names, an explanation that sounds as though it came from someone who has spent the week inside the problem. The review gets shorter. Not because everyone suddenly became lazy, though some of us had a head start, but because form is a signal. Our brains use signals to decide where attention is worth spending. AI can now produce that form in industrial quantities, for almost nothing. That changes the scale of the problem.

Researchers have a name for one version of this: epistemia, the illusion of knowledge that appears when surface plausibility replaces verification.1 The word is slightly forbidding, but the scene is familiar. We read a fluent answer, recognise its vocabulary, see a structure we trust, and stamp it “probably knows what it is talking about.” Does it? Is what it says true?

The argument over whether a model is intelligent or a stochastic parrot is worth having. I am happy to leave it to people who work on it seriously; in my hands it would become an exercise in style. I am interested in a more ordinary question: how do you tell an attractive answer from one you can make an important decision on? How do you tell the feature branch that will flood production with bugs from the one you ship and forget?

In a software team, certainty is not a starting point. At best, it is what a verification process earns. To decide, you need to know how a solution could break: which invariant it must preserve, which test should fail if you are wrong, which constraint you are taking for granted, and where the data behind the decision came from.

An answer is not reliable because it came from a person, a model, or some combination of the two. It becomes more reliable when it leaves a trail that can be checked. A small difference in a sentence; a large one in daily work. It moves the discussion from “this feels right” to “let’s see whether it holds.”

Plausibility has two effects. It can shorten the distance between doubt and false certainty. It can also give a doubt a form that others can work with. The decisive factor is the person using the tool. That person decides whether the result remains a hypothesis to test or becomes a shortcut around thinking.

Even measuring the value of these tools has become harder. METR tried to repeat an experiment on AI’s effect on software development and ran into a surprising problem: a growing share of developers did not want to work without AI in order to form the comparison group.2 That does not prove that AI always saves time or produces better code. It does show that the tool has entered work habits, and that assessing its effects now requires more careful methods.

One personal observation, not a statistic: when a company does not provide the tool, someone sometimes pays for it themselves and keeps quiet about it. Treating AI only as something to ban does not remove the phenomenon. It moves it out of sight.

That leaves companies with the harder job. They need to govern code that can look correct when it is not, alongside people who have already made these tools part of how they work. The answer is not to hide AI under the table. It is to make visible the step between a proposal and a decision.

The useful questions are almost boring: “What test would disprove this?”, “Which constraint are we assuming?”, “Where did this data come from?” We have not all become experts by magic. But people making decisions need to know that the burden of proof weighs more heavily than it used to, in software, law, or economics.

I do not expect AI to replace us in every part of the job. Producing proposals will keep getting cheaper; judging them, managing risk, and changing your mind will keep getting more expensive. Humans will remain the bottleneck. For now, and perhaps that is good news, they remain necessary too.

Footnotes

  1. Loru, Nudo, Di Marco, Santirocchi, Atzeni, Cinelli, Cestari, Rossi-Arnaud, and Quattrociocchi, The simulation of judgment in LLMs, PNAS (2025), doi: 10.1073/pnas.2518443122. ↩

  2. Becker, Rush, Cunningham, Rein, and Mahamud, We are Changing our Developer Productivity Experiment Design, METR (2026). ↩