Agentic AI for QA, building a Jira to Playwright pipeline with human approval
Software Testing Mentor and RCV Academy 
A single AI assistant helping one tester write one test is useful. Several agents working together, each handling one stage of a pipeline and handing off to the next, is a different thing entirely, and it's the direction QA automation is actually heading in 2026. This walks through building that kind of pipeline, from a raw Jira ticket to a reviewed, runnable Playwright test, with a human approval step that keeps it from running unsupervised.
Why one agent isn't enough
Ask a single AI assistant to go from a Jira ticket straight to a merged Playwright test, and you'll get something that mostly works but skips steps a careful tester wouldn't skip, since the model is trying to do requirements analysis, test design, and code generation all in one pass. Splitting that into separate agents, each with one narrow job, produces more reliable output, and it also gives you a natural place to insert a human checkpoint between stages rather than only at the very end.
The three stage pipeline
Stage one, the requirements agent. This agent reads the Jira ticket, including its description, acceptance criteria, and any linked tickets, and produces a structured test scenario in plain language, what should be tested, what the expected outcome is, and what edge cases the acceptance criteria implies even if they weren't spelled out explicitly.
Stage two, the code generation agent. This agent takes the structured scenario from stage one and writes it as an actual Playwright test, following your project's existing conventions, page object structure, naming patterns, the works. It has access to your codebase, so it can match the style of tests already in the repository instead of generating something stylistically foreign.
Stage three, the execution agent. Once a human has approved the generated test, this agent runs it, and if it fails, investigates whether the failure is a real bug, a flaky test, or a mistake in the generated test itself, then reports back with that classification rather than just a pass or fail.
Where the human approval gate goes
The most important design decision in this pipeline isn't which framework to use, it's where you insert human review. Put it between stage two and stage three, after the test is written but before it runs against anything real. This is exactly the pattern the Microsoft Agent Framework supports natively through what's called a human in the loop configuration, where the pipeline pauses and waits for explicit approval before taking a consequential action.
This matters because stage two, code generation, is where a subtle misunderstanding is most likely to slip through unnoticed, an agent that misread which field was required, or generated an assertion that's technically valid but tests the wrong thing. A human skimming the generated test before it runs catches that in seconds, far cheaper than debugging a mysterious failure after the fact.
What the handoff between agents looks like in practice
Using the Microsoft Agent Framework, a sequential pipeline structure lets you define this exact flow, each agent completing its stage and passing structured output to the next, with the framework itself managing state between them. A simplified version of the pattern looks like this conceptually.
requirements_agent -> structured_scenario
structured_scenario -> code_agent -> generated_test
generated_test -> [human review, approve or reject]
approved_test -> execution_agent -> run and classify result
Each arrow is a real handoff, not just a chained prompt, which means you can inspect, log, and debug what happened at each individual stage rather than treating the whole thing as one opaque process.
Handling the rejection path
Real pipelines need a path for when a human rejects the generated test, not just the happy path where everything gets approved. Build in a way to send rejected output back to the code generation agent along with the reviewer's specific feedback, rather than starting over from scratch. This mirrors how a human reviewer would hand back a pull request with comments rather than asking for a completely fresh attempt.
What this pipeline is not good at yet
Don't expect this to handle genuinely ambiguous requirements well. If the Jira ticket itself is vague or contradictory, the requirements agent will produce a scenario that reflects that ambiguity rather than resolving it, since resolving genuine ambiguity in a spec is still fundamentally a human judgment call, not something an agent should be guessing at unsupervised.
Common questions and queries you may have
Do I need the Microsoft Agent Framework specifically, or can I build this with something else?
The pattern works with several agent orchestration frameworks. The Microsoft Agent Framework is a reasonable starting point because its human in the loop support is built in rather than something you have to construct yourself.
How long does a pipeline like this take to build from scratch?
A working three stage version with a basic approval gate is realistic within a few days for someone comfortable with Python and familiar with your existing test framework. Refining the handoff quality between stages takes longer and benefits from real usage feedback.
Does this replace the need for testers to still write tests manually sometimes?
No. This pipeline handles the repetitive translation from ticket to test case well. Genuinely novel testing scenarios, exploratory testing, and anything requiring deep product context still benefit from a tester building the test directly.
RCV Academy's Agentic AI for QA and SDET course builds this exact pipeline step by step using the Microsoft Agent Framework, from a single two agent conversation up to the full three stage system described here.
Categories: : Agentic AI, AI, AI SDET, AI Tools, JIRA, Playwright
