Confidence Scores Are Not Test Results: Evaluating AI Features Beyond Pass or Fail

A test result is binary. A build is green or it is red. Confidence is not built that way, and treating a confidence score as if it were a pass or fail verdict is how a team ships an AI feature that sounds certain while being wrong.

What that percentage next to the answer actually measures

When an interface shows a confidence score next to an AI generated answer, it is usually derived from token level probabilities, often called logprobs. OpenAI's own cookbook explains that these values represent how likely the model considered each token it generated, given everything that came before it. That is a measure of how predictable the model's own wording was, not an independent check of whether the underlying claim is true.

This distinction matters because a model can be extremely fluent and confident while stating something false. In its own research on hallucination, OpenAI found that when researchers asked a widely used chatbot for the title of a specific person's PhD dissertation, it confidently produced three different answers across attempts, none of them correct. The same happened when asked for that person's birthday. The model was not hedging. It was wrong with full confidence, repeatedly. A tester relying on a displayed confidence number in either of those cases would have seen a high score attached to a wrong answer, with nothing in the interface signaling a problem.

The gap between stated confidence and actual accuracy has a name

Researchers call this gap calibration, and it has been measured directly. A study evaluating eleven popular language models on a range of reasoning tasks found that the models were, on average, overconfident, and that the overconfidence grew worse specifically on harder tasks, the exact situations where a team would most want an honest signal that the model was unsure. On the easiest tasks, several models were actually underconfident. There was no consistent direction a tester could simply correct for with one offset.

OpenAI's hallucination research explains part of why this happens. Standard training and evaluation reward a confident guess over an honest "I don't know," the same way a multiple choice test rewards guessing over leaving an answer blank. A model that hedges constantly scores worse on most benchmarks than one that guesses fluently, so the training process pushes toward fluent guessing.

Building an evaluation that does not mistake confidence for correctness

The ISTQB has already built a response to this into its certification content. The CT-GenAI syllabus treats evaluation as a distinct discipline from traditional pass or fail testing, covering criteria such as factual accuracy, relevance, and groundedness as separate, explicitly checked dimensions rather than folding them into a single score. If your team is building evaluation criteria for the first time, the Generative AI and AI Agents for QA and SDETs masterclass walks through setting these dimensions up in a real pipeline rather than treating them as an afterthought.

A workable evaluation checks correctness separately from confidence, using a ground truth answer or a retrieval source the output can be checked against, not the model's own stated certainty. It also tracks calibration itself as a metric over time, comparing how often high confidence answers turn out correct against how often low confidence answers do, since a model that is consistently overconfident on one category of question is a pattern worth flagging even when individual answers look fine. And it treats a hedge, an "I am not certain" or an explicit abstention, as a legitimate passing outcome rather than a failure, since penalizing honesty about uncertainty is exactly the incentive that produces confident wrong answers in the first place.

None of this needs to be complicated to start. A test suite can log the confidence score alongside the correctness verdict for every case, then periodically bucket results by confidence range and check the actual accuracy inside each bucket. If the ninety percent confidence bucket is only correct sixty percent of the time, that gap is the calibration problem made visible, and it is a far more useful thing to report to a team than a single aggregate pass rate that hides which answers were lucky guesses dressed up as certainty.

Frequently asked questions

If a confidence score is not reliable, why do AI products keep showing one?

Because it is easy to compute and it reads as informative, even though it mostly reflects how predictable the wording was rather than how true the claim is. A product can show it as one signal among several without treating it as the verdict on correctness.

Can a low confidence score still be trusted as a warning sign?

Somewhat. A low score sometimes does correlate with a harder or more ambiguous question. But the calibration research shows the relationship is inconsistent across models and task difficulty, so a low score is a hint worth investigating, not proof of an error, and a high score is not proof of the opposite.

Does this mean confidence scores are useless for testing?

No. They are useful as one input alongside correctness checks, especially for spotting patterns, like a model that is consistently overconfident on a specific category of question. The mistake is using confidence alone as the pass or fail gate.

How is this different from testing nondeterministic output in general?

Nondeterminism is about a model giving different valid answers to the same prompt across runs. Confidence miscalibration is about a model being wrong and not knowing it. A feature can be well calibrated and still nondeterministic, or deterministic and badly calibrated. They need separate checks.

If your team is deciding how to build correctness and calibration checks into an existing automation pipeline rather than starting from a blank page, the Agentic AI for QA and SDET course covers wiring these evaluation layers into a working multi agent workflow.