How to Test an AI Feature When the Output Is Never the Same Answer Twice

Ask a chatbot the same question twice and you can get two different, both defensible, answers. Traditional test design assumes that never happens. A test case has an expected result, the actual result either matches it or it does not, and that single comparison is the entire verdict. Feed that assumption an AI feature and it breaks on the first run.

The pass or fail assumption was never built for this

Most testers learned their craft on deterministic systems. Same input, same output, every time, and if it did not match, something was broken. Generative AI features do not work that way by design. A large language model predicts the next token from a probability distribution and samples from it, so the same prompt can legitimately produce different, equally correct, wording, structure, or even reasoning path across runs. That is not a bug to file. It is the mechanism.

The scale of the problem is bigger than most testers expect. A King's College London and University College London study tested ChatGPT on 829 coding problems across three benchmarks, generating five responses per prompt with identical instructions. The ratio of tasks where not a single one of those five responses produced the same test output was 75.76 percent on CodeContests, 51.00 percent on APPS, and 47.56 percent on HumanEval. Code generation is usually considered one of the more constrained, less creative AI tasks. If it is that unstable, free form chat or content generation features are not going to be more forgiving.

Lowering the temperature does not fix it

Most teams assume there is a setting for this: turn temperature down to zero and the model becomes deterministic. The same study tested that assumption directly and found it does not hold. Even at temperature zero, 43.64 percent of CodeContests tasks and 18.29 percent of HumanEval tasks still produced no equal output across five identical requests. OpenAI's own documentation confirms that a lower temperature narrows the sampling distribution toward the most likely tokens, it does not eliminate sampling. Hardware level nondeterminism in how GPUs execute floating point operations plays a role too, and that layer is outside any setting a tester can control.

Techniques built for output that keeps changing

The ISTQB has already formalized this as its own test design category. Chapter 3 of the CT-GenAI syllabus covers test design for nondeterminism specifically, including perturbation testing, scenario coverage across repeated runs, and metamorphic testing, where you check that a known transformation of the input produces a correspondingly sensible change in the output rather than checking for one exact answer.

Semantic comparison replaces exact string matching as the core assertion. Instead of checking that the output equals a fixed string, you check that it means the same thing as an expected answer, using an LLM as a judge against defined criteria. DeepEval's documentation describes this as G-Eval, a framework that scores output against natural language criteria using chain of thought reasoning rather than a literal match, precisely because a literal match assertion fails constantly against legitimate output variation.

Repetition turns single runs into a distribution you can reason about. DeepEval also ships a repeats option that reruns the same test case multiple times so a team can look at a pass rate and a variance across runs, not a single pass or fail verdict from one sample. That shift, from one comparison to a distribution of comparisons, is the core adjustment nondeterminism testing asks of a tester. If your team is building this kind of evaluation pipeline, the Generative AI and AI Agents for QA and SDETs masterclass covers setting up evaluation frameworks like this from the ground up.

What this looks like in a real test suite

Run each AI test case more than once before calling it a pass. A single green result tells you almost nothing about a feature whose output varies by design. Three to five runs per case, tracked as a pass rate, gives you something you can actually trust.

Write assertions against meaning, not characters. A test that checks for an exact string will fail constantly on a feature that is working correctly, which trains a team to ignore its own test suite. Semantic similarity thresholds and criteria based scoring hold up better against legitimate variation.

Separate content variation from a real defect. A support bot that answers a question correctly in two different phrasings is fine. A support bot that gives a factually different answer to the same question is not. The first case is nondeterminism working as intended. The second is a defect that nondeterminism happens to be hiding, and the two need different assertions to tell apart.

Frequently asked questions

Is nondeterminism the same problem as a flaky test?

No. A flaky UI test fails for reasons unrelated to the product, like a race condition or a slow load. Nondeterministic AI output is the product behaving correctly while producing a different, still valid, response each time. The fix for one is stabilizing the test. The fix for the other is testing a distribution instead of a single output.

Does setting temperature to zero make testing simpler?

It reduces variation somewhat, which can help, but it does not remove the need for semantic assertions or repeated runs. Treat a low temperature as one mitigation among several, not a substitute for testing the output's meaning.

Can traditional automation tools like Selenium or Playwright test AI features at all?

Yes, for anything deterministic around the AI feature, like whether a response renders, loads within a time limit, or triggers the right UI state. They are the wrong tool for judging whether the AI generated response itself is correct, which needs a semantic or LLM based evaluation layer instead.

How many times should a test case be rerun to trust the result?

There is no universal number. Three to five runs is a common starting point for catching obvious instability without making a suite prohibitively slow, and teams testing higher risk features often run more and track variance over time rather than picking one fixed number and stopping there.

If your team is putting together a test strategy for a feature that will never give the exact same answer twice, the ISTQB CT-GenAI certification course walks through the nondeterminism test design techniques this post only had room to introduce.