Temperature Settings and Determinism for AI Testing

What Temperature Actually Controls

Every time a language model generates a token, it produces a probability distribution over its entire vocabulary. The word “test” might score 0.32, “verify” might score 0.18, “check” might score 0.14, and thousands of other tokens share the remainder. Temperature is the parameter that reshapes that distribution before the model picks.

At temperature=0, the model applies greedy decoding: it always picks the highest scoring token. The distribution collapses to a single choice. At temperature=1.0, the distribution is used without modification. At values above 1.0, the distribution flattens, making lower probability tokens more competitive. At values below 1.0, the distribution sharpens, concentrating probability mass on the tokens that already rank highest.

For QA and testing workflows, this matters because output variability is not always desirable. When you ask a model to generate test cases from a specification, you generally want the same specification to produce the same cases each time. When you ask it to explain a defect, you want the explanation to stay consistent across team members who run the same prompt. Temperature is the primary dial for controlling that variability.

The Reproducibility Gap

Here is the part that catches teams off guard: setting temperature to zero does not guarantee identical outputs.

A 2024 study tested five large language models across eight tasks over ten runs each, all with temperature=0 and fixed seeds. Accuracy varied by up to 15 percentage points across runs. The gap between best and worst runs reached 70 percentage points in some configurations. Mixtral 8x7B on college mathematics problems scored 75% on its best run and 3% on its worst, a gap of 72 points on the same task with the same settings. GPT-4o on the same benchmark ranged from 88% at best to 44% at worst, with a median of 69% (Atil et al., “LLM Stability: A detailed analysis with some surprises,” arXiv:2408.04667, 2024).

The cause is not randomness at the model level. It is infrastructure. Modern serving systems use continuous batching, chunked prefill, and prefix caching to serve many requests concurrently and efficiently. These optimizations affect floating point operation ordering, which in turn affects accumulation errors at the token level. The net effect is that the same prompt can produce different outputs depending on what else the server was processing at the time.

A separate 2025 study tested GPT-4o mini, GPT-4o, Claude 3.5 Sonnet, and LLaMA 3.2 90B Vision on 70 Java commits with five repetitions at temperature=0. None of the models produced fully consistent outputs. The authors note that practitioners should treat claimed determinism with caution when making production decisions based on AI output (Ferdaws et al., “Measuring Determinism in Large Language Models,” arXiv:2502.20747, 2025).

The Seed Parameter and Its Limits

OpenAI introduced a seed parameter specifically to address reproducibility concerns. The intent is that a fixed seed with temperature=0 should produce “mostly” consistent outputs. The documentation deliberately uses “mostly” rather than “always.”

The mechanism involves an additional signal called system_fingerprint. When a response includes a system_fingerprint, it records the backend configuration state at the time of generation. If that fingerprint changes between two calls, the infrastructure has changed and the outputs may differ even with identical seeds. Teams using the seed parameter for reproducibility need to track the fingerprint alongside the seed.

The practical implication is that the seed parameter is useful for reducing variability, not eliminating it. It is a tool for reducing variance, and the documentation is explicit about that limit. For audit purposes or test case archiving, saving the actual output is more reliable than assuming a prompt plus seed will reproduce it.

Choosing Temperature for Testing Tasks

The right temperature depends on what the task requires. A few working patterns for QA teams:

Temperature=0 for regression testing prompts. When a prompt is part of a test suite and you want to catch changes in model behavior over time, temperature=0 minimizes variation across runs. Accept that some variation is still possible, and treat unexpected output changes as worth investigating rather than dismissing.

Temperature=0.2 for test case generation. A small amount of variability helps the model explore the specification without producing radically different outputs on each run. The output stays structured and mostly predictable while still generating cases that cover different angles of the requirement.

Temperature=0.7 for exploratory testing prompts. When asking a model to think about edge cases, unusual user paths, or failure modes you might not have considered, higher temperature encourages more varied outputs. Run the prompt several times and look at what differs across runs.

Temperature=1.0 or above for brainstorming and ideation. If you want a model to generate test ideas without being constrained by the most obvious paths, higher temperatures produce more diverse outputs. These are starting points, not finished test cases.

The table in the inline image above maps these levels to output characteristics and use cases as a quick reference.

Making Reproducibility Work in Practice

Given that true determinism is not available from models served through cloud APIs, teams can adopt a few approaches that make reproducibility practical.

Store outputs alongside prompts. For any test case or criterion that will be reused, save the actual model output rather than relying on running the prompt again to reproduce it. Treat the output as the artifact, version it, and review changes to it the same way you would review changes to any other test asset.

Use temperature=0 as a baseline, not a guarantee. Set temperature=0 for all production prompt calls where consistency matters. Accept that some variation will occur and build review steps around that assumption. Monitor outputs over time and flag significant changes.

Run multiple passes for high stakes generation. When a model is generating test cases that will drive a regression suite, run the generation three to five times at temperature=0 and compare outputs. Cases that appear across all runs are stable. Cases that appear only sometimes are worth human review before inclusion.

For teams building these controls into structured workflows, the Generative AI and AI Agents for QA SDET Masterclass covers prompt design patterns and output validation techniques for production QA pipelines.

Teams looking to automate the comparison and review steps described above will find the architectural patterns in Agentic AI for QA and SDET: Build Autonomous Test Automation, including how to build stable generation pipelines around models that do not guarantee exact output reproducibility.