Semantic Understanding: How AI Agents Process Tests

What the Agent Actually Sees

When you type “verify the checkout flow works correctly,” the agent does not read that sentence the way you do. It does not retrieve a mental model of checkout, correctness, or flow. What it receives is a sequence of numerical tokens: subword pieces produced by breaking your text into fragments that appear frequently in the training corpus. “Checkout” becomes one or two tokens. “Correctly” becomes another. Each token maps to a position in a vector space with many dimensions, shaped by billions of training examples.

That vector representation drives every subsequent decision the agent makes. When two test commands land close together in that space, “confirm order submission succeeds” and “check that placing an order works,” the agent treats them as semantically similar, even if their surface wording shares almost nothing. When a command contains an unfamiliar identifier, say a component name specific to your codebase, the model has no vector for it. It falls back on adjacent tokens and infers meaning from context. That inference is where most of the interesting failures happen.

Understanding this pipeline does not require deep linear algebra. It does require knowing roughly where the failure points are so you can write instructions the agent has a fair chance of acting on correctly.

The Pipeline From Command to Action

Every AI agent that takes a natural language test instruction and produces an action runs a version of the same pipeline. The instruction is tokenized, mapped to embeddings, passed through attention layers that weight the relationships between tokens, and decoded into a decision: a tool call, a selector, a code block, or an assertion.

The attention mechanism is what gives transformers their ability to handle context. A token like “it” in “click the button and verify it is enabled” gets attended to the token “button” through learned attention weights, so the model resolves the reference without being told explicitly. This works well for common constructions. It breaks down when the sentence structure is unusual, when the referent is ambiguous, or when the identifier names do not appear in training data.

A 2025 paper by Hussain et al. evaluated GPT-4o and StarCoder on test generation from bug reports using a framework built around Bloom’s taxonomy (Hussain et al., “Test Case Generation from Bug Reports via Large Language Models: A Cognitive Layered Evaluation Framework,” arXiv:2510.05365, 2025). Under identifier mutations, where component names were altered to names not in the training corpus, performance dropped by more than 60%. The same models showed partial robustness to rephrasing and translation, but fell sharply when the names they needed to act on were unfamiliar. The study also found that structured elements like test method names and code context mattered far more than narrative descriptions for successful generation.

Where Context Retrieval Changes the Outcome

The single biggest improvement available to QA teams working with AI agents is not prompt length or model size. It is grounding the agent in the actual structure of what it is supposed to test.

A January 2026 paper by Vali describes a retrieval augmented generation system for producing Selenium test scripts (Vali, “Autonomous QA Agent: A Retrieval-Augmented Framework for Reliable Selenium Script Generation,” arXiv:2601.06034, 2026). The system ingests project documentation in Markdown, PDF, and HTML into a vector database. Before generating any test code, it retrieves the relevant page’s DOM structure as additional context. The result on 20 e-commerce scenarios: 100% syntax validity and 90% execution success, against 30% for standard LLM generation without retrieval. The improvement is not the model becoming smarter. It is the model being given the identifiers it needs before it has to guess them.

This is the practical implication of how embeddings work. A model that has never seen your component named “AddToCartBtn” will try to infer it from surrounding text. A model that retrieves the DOM before generating code sees that identifier in context and can match it precisely. Retrieval does not fix the semantic processing pipeline. It feeds the pipeline better inputs.

What Good Prompt Structure Does

The same principle applies to how you write instructions. If the agent’s semantic processing is driven by token proximity and learned attention patterns, the structure of your prompt shapes the quality of the output before any retrieval happens.

Research on GUI test execution using LLM agents found that when a test step does not match a controlled grammar, a restricted natural language the system can reason about, the agent rewrites it into atomic steps (Salva, “Reliable Execution of Natural Language Test Cases for GUI Applications using LLM Agents,” Software Quality Journal, 2026). Llama3.3 70B running this approach produced a 0% false negative rate on navigation steps, compared with far higher error rates on the naive baseline. The system treated ambiguous steps as inconclusive rather than attempting execution and potentially producing a wrong result. That conservative handling, failing to inconclusive rather than failing silently, is only possible when the input format is structured enough to detect ambiguity.

The failure modes that a structured prompt prevents are not trivial. A formative study by Mahmud and Kandogan examined 236 analytical intents processed by agentic systems and found 153 recurring failures despite successful workflow generation and execution (Mahmud and Kandogan, “Exploring the Semantic Gap in Agentic Data Systems: A Formative Study of Operationalization Failures in Analytical Workflows,” arXiv:2607.00828, 2026). Five recurring failure classes emerged: comparative grounding, where the agent could not resolve what “better” or “higher” meant against the data; process reasoning; quantitative reasoning; role confusion; and policy grounding. For QA instructions, the equivalent failures are assertions that pass the wrong condition, selectors that match the wrong element, and test steps that execute against the wrong state.

Writing Instructions the Agent Can Process Reliably

The research points to a small number of concrete practices.

Use the exact identifier names from your application. Do not paraphrase component names or describe them generically. “Click the submit button” may resolve correctly. “Click the form submission control” will more often require an inference step that can fail.

Provide DOM or API context when the agent needs to select elements or call endpoints. Retrieval handles this automatically in a properly configured agent. When retrieval is not available, include the relevant fragment in the prompt itself.

Keep each step atomic. The attention mechanism handles references within a sentence reasonably well. Instructions that chain multiple conditions, “click the button and then verify the modal opens and check the error message,” force the model to track multiple state transitions, and the later conditions frequently receive weaker attention.

Treat inconclusive verdicts as information. When an agent marks a step inconclusive rather than passing or failing it, that is the model signaling that the instruction was ambiguous at the semantic level. Rewrite those steps before treating the verdict as a meaningful result.

For teams building structured test prompts and wanting to understand how LLMs interpret instructions at the architecture level, the Generative AI and AI Agents for QA SDET Masterclass covers transformer fundamentals and how they apply to agent design for QA workflows.

Teams building retrieval augmented pipelines for test generation, using vector search against DOM documentation, API specs, or test history, will find the patterns and implementation covered in LangChain for QA: Build RAG, Agentic RAG and AI Chatbots.