Semi-Autonomous vs Fully Autonomous AI Agents

Why the Question Matters More Than the Hype

Most conversations about AI agents collapse the choice into a binary: either the agent does everything or a human does. That framing is wrong, and it leads to bad deployments. The real decision is not how much autonomy the agent has in theory. It is how much autonomy is appropriate for a specific task, with a specific cost of error, against a specific team’s capacity to monitor and intervene.

Researchers at the University of Washington and University of Chicago recently proposed a framework with five levels for thinking through this decision (Feng, McDonald, and Zhang, “Levels of Autonomy for AI Agents,” arXiv:2506.12469, 2025). Their key insight is that autonomy is a design choice, not a model capability. A highly capable agent can be configured to consult the user frequently. A weaker one can be given wide latitude in a bounded environment. Capability and autonomy are independent variables.

The five levels run from Operator, where the user controls every action and the agent assists only when invoked, through Collaborator and Consultant, where control is shared or delegated with frequent input, to Approver, where the agent runs independently and the user gates only consequential actions, and finally Observer, where the agent executes everything and the user can only monitor logs and throw an emergency stop. These names describe the role the user plays, not a specification of the agent’s technical architecture.

What Goes Wrong at the Wrong Autonomy Level

Running an agent at a higher autonomy level than the task warrants produces failure modes that differ in kind from errors in a single step. At ASE 2025, Lu, Li, and Huo reported that across 204 runs of three agent frameworks with open source code on 34 programmatic tasks, approximately 50% of runs failed. They identified 19 failure causes across three phases: task planning, task execution, and response generation (Lu, Li, and Huo, “Exploring Autonomous Agents: A Closer Look at Why They Fail When Completing Tasks,” arXiv:2508.13143, ASE 2025).

Two failure modes are worth calling out specifically. The first is the failure to refine: an agent that encounters a failed subtask and repeats the same approach rather than changing strategy, iterating until it hits the round limit. The second is context window exhaustion: in longer runs, earlier observations fall out of the active context, and the agent loses access to information it needs to complete the current step. Both of these emerge from running at high autonomy. A human reviewing the output at each step would catch both early.

A separate study of how developers actually oversee agents in practice found that the most effective control was applied before the agent started, not during the run (Dhanorkar, Passi, and Vorvoreanu, “Human Oversight of Agentic Systems in Practice,” arXiv:2606.05391, 2026). Developers used task scoping and deny lists to keep the agent from acting outside an intended boundary, rather than watching every step. One developer described the strategy plainly: keeping tasks small made monitoring during the run largely unnecessary.

Errors that go uncaught at the wrong autonomy level also propagate. A Replit incident in which an agent deleted a production database illustrates how fast an irreversible action can cascade. Navneet and Chandra, reviewing autonomous software engineering failures, identified irreversible actions, vulnerability inheritance, and the absence of rollback protocols as the principal structural risks when agents run with broad autonomy (Navneet and Chandra, “Rethinking Autonomy: Preventing Failures in AI-Driven Software Engineering,” arXiv:2508.11824, 2025).

The Framework in Practice for QA Teams

For testing workflows, the five levels translate into a practical decision tree.

Tasks with narrow scope, good documentation, and low cost of error are strong candidates for Level 4 autonomy (the Approver pattern). The agent runs test generation, locates selectors, and writes assertions. A human sees the output only when the agent hits a blocker or needs credentials. Autonomous QA research showing 90% execution success on e-commerce scenarios with retrieval augmented generation support sits squarely in this zone.

Tasks involving assertions on novel application states, integrations with production systems, or security boundaries belong at Consultant or Collaborator level (Level 2 or 3). The agent proposes and drafts. A human reviews the draft before it runs.

Tasks where the cost of a missed defect is high, say a payment calculation or an access control boundary, belong at Operator level (Level 1), at least for the assertion step. The agent may help draft the test, but the human writes the assertion and signs off before any run touches production data.

What Real Usage Data Shows

Anthropic’s analysis of autonomous agent usage in production found that human involvement dropped from 87% of tool calls on straightforward tasks to 67% on complex ones, and that roughly 0.8% of actions across the dataset were irreversible (Anthropic, “Measuring AI Agent Autonomy,” 2026). Among experienced Claude Code users, rates of full approval exceeded 40%, while interrupt rates rose from about 5% for newer users to 9% for experienced ones. The authors read the rising interrupt rate as active monitoring rather than disengagement: experienced users let agents run further but also watch more carefully.

More autonomy paired with more attentive oversight is the practical pattern the research points to. Experienced practitioners are developing judgment about which actions warrant a pause and which do not, not working toward full automation without oversight.

For QA teams, the action signal from this data is calibration. Start agents at a lower autonomy level than seems necessary. Measure the interrupt and rejection rates. Raise the autonomy level incrementally when the data supports it, not because the model seems capable enough. The cost of running an agent at Level 2 when Level 4 is technically achievable is a slightly slower pipeline. The cost of running at Level 4 when Level 2 was appropriate can be a defect in production.

Designing Autonomy Into the Workflow

A few structural choices make it easier to operate agents at the right level without constant renegotiation.

Define the reversal boundary explicitly. Before any agent task begins, the team should identify which actions in that task are reversible and which are not. Writing a test file is reversible. Running a test against a production endpoint that triggers a payment state change may not be. The reversal boundary determines the lowest safe autonomy level for that task.

Build scope constraints before approval gates. The developer oversight research found that narrow task scope was more effective than watching every step. A prompt scoped to a narrow task prevents a wider range of failure modes than an approval gate inserted after the agent has already started down an incorrect path.

Track what gets interrupted. An interrupt or rejection during an agent run is a signal, not only a correction. A team that logs what it interrupts will find patterns. Those patterns are the inputs for either raising the autonomy level when the agent is reliably right, or restricting the task scope when the agent is reliably wrong in the same way.

For teams building agentic QA workflows with appropriate oversight boundaries, the Agentic AI for QA and SDET: Build Autonomous Test Automation course covers agent architecture, task scoping, and how to design approval gates into a test pipeline.

Teams working through the conceptual foundations first, including the agent module architecture and patterns like ReAct, will find the Generative AI and AI Agents for QA SDET Masterclass covers the progression from LLM fundamentals through orchestration patterns and oversight design.