Self Healing Tests Explained: What They Actually Fix and What They Quietly Hide

Google's own engineers spend between 2 and 16 percent of their compute budget re running tests that failed for no real reason, according to the company's own research paper on continuous integration testing, which also found that almost 16 percent of their tests show some level of flakiness. That is the itch self healing test tools are built to scratch: stop burning time and compute on failures that have nothing to do with the product. It is a real problem. The fix just is not as clean as the marketing suggests.

Self healing is not one technique wearing different labels

The term covers at least three different mechanisms, and mixing them up leads to the wrong expectations.

Locator fallback is the oldest version. Katalon Studio's self healing documentation describes it plainly: when the default locator cannot find an object, Katalon tries other pre configured locators, such as an alternate XPath, CSS selector, or image match, associated with that same object. If one of them works, the test keeps running and Katalon suggests swapping in the locator that succeeded.

Visual comparison is a different approach entirely. Applitools built its Eyes platform around what it calls Visual AI, a perceptual comparison engine that judges a screenshot the way a person would rather than pixel by pixel, according to Applitools's own documentation. Applitools also offers an Execution Cloud that, per that same documentation, heals tests that fail because of small locator changes, layering the visual approach on top of the older locator repair idea.

LLM reasoning is the newest layer, and it is closer to judgment than pattern matching. Playwright's healer agent, covered in the official Playwright documentation, inspects a failing test, proposes a specific patch such as a new locator or an adjusted wait, and reruns the test to confirm the fix. It is a more capable process than a fixed list of fallback locators, and it comes with a correspondingly bigger blind spot.

What each approach is genuinely good at

Locator fallback earns its keep on cosmetic churn. A button's id attribute changes during a refactor, a wrapper div gets added around a form field, a class name gets renamed for a styling pass. None of that is a real defect, and a fallback locator finds the same element under its new fingerprint in milliseconds.

Visual comparison catches the opposite category well. It flags a layout shift, a missing element, or a broken style that a locator based test would sail right past, because the locator technically still resolves to something on the page even when that something looks wrong.

LLM based healing handles the messy middle. It can reason about a renamed button that also moved position, propose a wait adjustment for a slow loading component, and explain what changed in a way a person can review quickly. If you want to build this kind of pipeline yourself rather than adopt it as a packaged feature, the Agentic AI for QA and SDET course walks through building the review and approval steps around exactly this kind of agent.

What all three quietly hide

Every one of these mechanisms shares the same failure mode: a healer that finds a plausible replacement is not the same as a healer that confirms the application still works correctly. A checkout button that silently loses its tax calculation but keeps the same text and rough position can get "healed" by a locator fallback with no idea the underlying logic broke. A visual AI baseline that gets re approved without a careful look can bake a real regression into the new normal. An LLM healer that patches a locator to make a test pass again has not verified that the flow it is testing still does what it is supposed to do, only that something now responds where the original locator failed.

This is not a hidden flaw the vendors are covering up. Playwright's own documentation for the healer agent is explicit that a patch is a proposal, not a guarantee, and that human review belongs in the loop before that patch ships. The risk is not that the tools lie about what they do. It is that a team under deadline pressure treats a healed test the same as a test that never broke.

How to use self healing without losing the signal

Keep the failure log even after a heal succeeds. If a locator needed fixing three times in two weeks, that element is unstable for a reason worth investigating, not just re patching again.

Separate cosmetic heals from structural ones. A renamed CSS class is routine. A locator that now resolves to a different type of element than before, or a visual diff on a payment flow, deserves a person's eyes before it gets accepted.

Do not let a healed test skip code review. The commit that updates a locator or approves a new visual baseline should go through the same review path as any other change to the suite, even if the origin was automatic.

Frequently asked questions

Does self healing replace the need to maintain test locators well in the first place?

No. Every vendor's own documentation points to good locator strategy, stable ids and accessible attributes over brittle CSS paths, as the first line of defense. Self healing is a safety net for the cases that slip through, not a substitute for writing resilient selectors.

Can a self healed test hide a real bug from a code review too?

Yes, if the healed locator or approved baseline gets merged without anyone checking why the original one failed. The healing step fixes the symptom. Someone still has to ask whether the underlying cause was cosmetic or a real defect.

Is visual AI testing worth adopting if a team already uses locator based self healing?

The two catch different things. Locator fallback keeps a test running through cosmetic DOM changes. Visual comparison catches layout and rendering regressions that a resolvable locator would miss entirely. Teams testing anything visually sensitive typically want both.

How do I know if self healing is masking problems in my own suite?

Track how often each test gets healed over time. A test that heals once after a redesign is normal. A test that heals every few runs on the same element is telling you something is unstable, and repeated healing is treating a symptom that keeps coming back.

If your team is deciding how deep to build AI assisted maintenance into a suite versus how much to keep under manual review, the Master Playwright course covers the healer workflow end to end, including the review habits that keep a green build meaningful.