The most dangerous AI is the one that is almost never wrong
If an AI tool failed every day, no one would trust it. The problem starts when it gets things right. It gets it right once, ten times, a hundred, and at some point whoever reviews it stops doing so carefully. They glance at it, see that everything looks fine, and approve. That is the day the error gets through.
It happened in New York in 2023. Two lawyers filed a brief in federal court citing cases that did not exist. ChatGPT had generated them, with names, dates and quotations that looked real. The judge sanctioned them for signing without checking.
What interests me about this case is not the tool’s mistake. It is that one of them had been practicing for about thirty years and felt no need to check anything. Experience does not protect against this error. Sometimes it makes it easier, because someone who has seen a lot trusts their instinct that something “sounds right.”
Why we stop checking
Research has a name for this: automation bias. Raja Parasuraman and Dietrich Manzey, who spent years studying how people supervise automated systems, found a consistent pattern. The more reliable a system seems, the less attention we pay to it, and this happens to novices and experts alike. Practice alone does not fix it. Trust is not a character flaw. It is how the brain saves effort.
An experiment from 1948 explains it well. Norman Mackworth asked radar operators from the British Royal Air Force to watch a clock and report when the hand made a double jump, which happened only occasionally. Within half an hour, the number of jumps they caught had already dropped noticeably. The rarer the error, the harder it is to see.
AI puts us in exactly that position. A tool that is almost always right turns us into Mackworth’s operator: waiting for an error that hardly ever appears.
Aviation went through this first. Autopilot flies more precisely than any person, and pilots began to delegate too much. In 2013 the FAA, the U.S. aviation authority, issued a safety alert asking airlines to have their crews fly more by hand, because they were losing the practice they needed to take control when the system failed.
The risk almost no one mentions
There is a lot of talk about keeping a human in the loop. I apply it in all my work. But it has an uncomfortable side.
Researcher Madeleine Clare Elish called it the moral crumple zone, after the part of a car designed to collapse and absorb the impact. When an automated system fails, the blame lands on the human who was “in charge,” even if in practice they had neither the time, the information nor the training to catch the error.
In a company it looks like this: someone is asked to approve forty AI-generated reports a day. On paper, there is a human in the loop. In reality, that person is where the blame will land.
That is why, for me, the question is not whether a human is reviewing, but whether that human has a real chance of finding the error.
How I apply it
I learned this long before AI existed, preparing budgets and month-end closes: a number no one verified ends up, sooner or later, in a bad decision. Today I organize it around four criteria.
The first is that the level of review depends on what is at stake. An internal summary of trends can be spot-checked. A projection that will justify a loan, a hire or an investment is reviewed in full, figure by figure, against the original source. And there are decisions AI never touches: approving payments, deciding on credit or committing capital.
The second is that the reviewer has to know more than the tool about that subject. A reviewer who does not understand the analysis cannot catch the error, only sign it. And they need time: if someone reviews in one minute what should take twenty, that review is not real.
The third is seeding errors. From time to time I deliberately introduce a wrong figure or a nonsensical assumption into a report prepared with AI, and see whether anyone catches it. It is what flight simulators do when they trigger a failure. If the error gets through, I know the problem is not the tool but the process. The point is not to catch anyone out; it is to measure whether the control works.
The fourth is keeping count of corrections and, now and then, doing the work by hand. If in three months no one has corrected anything the AI produced, most likely no one is reading it. And if the team stops doing the analysis on its own, it loses the judgment it needs to notice when something is wrong. It is the same logic as the FAA’s push for manual flying.
What it costs
All of this has a price: reviewing properly takes time and gives back part of the speed AI promised. Some prefer not to pay it. My experience is that AI’s time savings are only real once you subtract the cost of the errors it lets through. A report prepared in five minutes that led to a bad decision was the most expensive one of all.
In the Prevuelo™ Method (prevuelo is Spanish for “preflight”) I sum it up with the cockpit rule: technology helps you fly, but the captain stays in command and has to keep knowing how to fly.
If your company already uses AI, it is worth asking when someone last found an error in its work. If no one remembers, the problem is probably not the tool.
MR³ Index
Before your next big decision, measure where your company stands in 5 minutes.