---
title: "Is there still room for doubt?"
description: "When a polished answer arrives without verification, our job is to decide what has actually earned our trust."
lang: en
canonical: https://potenza.dev/writing/is-there-still-room-for-doubt/
alternate:
  it: https://potenza.dev/it/blog/il-dubbio/index.md
author: "Emilio Potenza"
updated: 2026-08-01
---

> Site index
> Markdown pages are indexed at https://potenza.dev/llms.txt

A good answer can be dangerous when you do not quite know where it came from.
You stop arguing with it. It becomes settled before it has become true.

The same thing happens with code. A neat diff, sensible names, an explanation
that sounds as though it came from someone who has spent the week inside the
problem. The review gets shorter. Not because everyone suddenly became lazy,
though some of us had a head start, but because form is a signal. Our brains use
signals to decide where attention is worth spending. AI can now produce that
form in industrial quantities, for almost nothing. That changes the scale of
the problem.

Researchers have a name for one version of this: *epistemia*, the illusion of
knowledge that appears when surface plausibility replaces verification.[^pnas-epistemia]
The word is slightly forbidding, but the scene is familiar. We read a fluent
answer, recognise its vocabulary, see a structure we trust, and stamp it
“probably knows what it is talking about.” Does it? Is what it says true?

The argument over whether a model is intelligent or a stochastic parrot is
worth having. I am happy to leave it to people who work on it seriously; in my
hands it would become an exercise in style. I am interested in a more ordinary
question: how do you tell an attractive answer from one you can make an
important decision on? How do you tell the feature branch that will flood
production with bugs from the one you ship and forget?

In a software team, certainty is not a starting point. At best, it is what a
verification process earns. To decide, you need to know how a solution could
break: which invariant it must preserve, which test should fail if you are
wrong, which constraint you are taking for granted, and where the data behind
the decision came from.

An answer is not reliable because it came from a person, a model, or some
combination of the two. It becomes more reliable when it leaves a trail that
can be checked. A small difference in a sentence; a large one in daily work.
It moves the discussion from “this feels right” to “let's see whether it
holds.”

Plausibility has two effects. It can shorten the distance between doubt and
false certainty. It can also give a doubt a form that others can work with. The
decisive factor is the person using the tool. That person decides whether the
result remains a hypothesis to test or becomes a shortcut around thinking.

Even measuring the value of these tools has become harder. METR tried to repeat
an experiment on AI's effect on software development and ran into a surprising
problem: a growing share of developers did not want to work without AI in order
to form the comparison group.[^metr-2026] That does not prove that AI always
saves time or produces better code. It does show that the tool has entered work
habits, and that assessing its effects now requires more careful methods.

One personal observation, not a statistic: when a company does not provide the
tool, someone sometimes pays for it themselves and keeps quiet about it.
Treating AI only as something to ban does not remove the phenomenon. It moves
it out of sight.

That leaves companies with the harder job. They need to govern code that can
look correct when it is not, alongside people who have already made these tools
part of how they work. The answer is not to hide AI under the table. It is to
make visible the step between a proposal and a decision.

The useful questions are almost boring: “What test would disprove this?”,
“Which constraint are we assuming?”, “Where did this data come from?” We have
not all become experts by magic. But people making decisions need to know that
the burden of proof weighs more heavily than it used to, in software, law, or
economics.

I do not expect AI to replace us in every part of the job. Producing proposals
will keep getting cheaper; judging them, managing risk, and changing your mind
will keep getting more expensive. Humans will remain the bottleneck. For now,
and perhaps that is good news, they remain necessary too.

[^pnas-epistemia]: Loru, Nudo, Di Marco, Santirocchi, Atzeni, Cinelli, Cestari, Rossi-Arnaud, and Quattrociocchi, [*The simulation of judgment in LLMs*](https://arxiv.org/abs/2502.04426), *PNAS* (2025), doi: [10.1073/pnas.2518443122](https://doi.org/10.1073/pnas.2518443122).
[^metr-2026]: Becker, Rush, Cunningham, Rein, and Mahamud, [*We are Changing our Developer Productivity Experiment Design*](https://metr.org/blog/2026-02-24-uplift-update/), METR (2026).