Shift Right Is Back: Why Production Testing Is Getting Serious Again

For years, saying "we test in production" got you laughed out of a standup. It sounded like a euphemism for not testing at all. That reputation is fading, and it is fading because staging environments have quietly lost their ability to represent what actually happens when real traffic hits a real system.

Staging was never going to catch everything

This is not a new problem, just a more visible one. NIST's 2002 Planning Report on the economic impacts of inadequate software testing infrastructure estimated that inadequate testing infrastructure cost the US economy 59.5 billion dollars annually, and found that more than half of software bugs were not caught until downstream in the development process, well past the stage where a staging environment could have caught them cheaply. That finding predates microservices, containers, and the scale most systems run at today, and the underlying problem it describes has only gotten harder to avoid.

A staging environment can approximate load, but it cannot replicate real user behavior, real data shapes, real third party latency, or the exact combination of services that happen to be deployed together at any given moment in a system built from dozens of independently released microservices. Some classes of failure are only observable once real traffic is flowing through the real system.

What shift right actually means, beyond a slogan

The term gets used loosely, so it is worth being specific about the practices behind it, rather than treating "testing in production" as one vague idea.

Observability is the foundation the rest depends on. OpenTelemetry, the vendor neutral open source standard now used to instrument traces, metrics, and logs across languages and platforms, defines observability as the ability to understand a system's internal state by examining its outputs. That is a meaningfully different goal than traditional monitoring, which only tracks metrics you already decided were important. Observability is built for the failure you did not know to look for.

Chaos engineering is the deliberate, disciplined version of breaking things on purpose. The Principles of Chaos Engineering, the community manifesto that grew out of practices developed at Netflix, defines it as experimenting on a system to build confidence in its ability to withstand turbulent conditions in production, and explicitly recommends running these experiments in production itself, with the smallest possible blast radius, rather than treating a staging approximation as good enough evidence.

Feature flags and canary releases turn deployment into a controlled experiment instead of an all or nothing event. Exposing a change to one percent of real traffic, watching what happens, and rolling back in seconds if it does not, catches the specific category of failure that only shows up under genuine production conditions and genuine user behavior.

The metrics that turned this into a measurable discipline

Shift right stopped being a fringe practice partly because it became something organizations could actually measure and benchmark against each other. DORA's research, the long running DevOps research program now associated with Google Cloud, restructured its core metrics in 2024 to include change failure rate and rework rate alongside deployment frequency and lead time, treating a team's ability to detect and recover from a bad production change as a first class signal of delivery performance, not an afterthought bolted onto speed metrics. Reliability, tracked through service level objectives and indicators grounded in real production behavior, sits alongside these as a deliberate acknowledgment that shipping fast only matters if the system stays trustworthy once it ships.

That shift in what gets measured is part of why shift right stopped sounding like an excuse. A team with a fast, well instrumented recovery process and a low change failure rate is not skipping testing. It is running a different kind of testing, one that produces evidence a staging environment cannot.

What this actually changes for the person doing the testing

The skill set shifts from writing test cases to reading signals. A tester working this way needs to read a distributed trace well enough to tell whether a slow request is a real regression or normal variance, define what "steady state" means for a specific service before a chaos experiment runs against it, and know which metric a feature flag rollout should be watched against before flipping it past one percent of traffic. None of that replaces pre release testing. It sits alongside it, covering the specific category of defect that only exists once real traffic and real data are involved.

Frequently asked questions

Does shift right testing mean skipping testing before release?

No. It is a complement to pre release testing, not a substitute for it. Unit tests, integration tests, and staging validation still catch what they are good at catching. Shift right exists because some failures genuinely only appear under real production conditions.

Is chaos engineering too risky for a smaller team to attempt?

The core principle scales down. A small, deliberate experiment with a tightly limited blast radius and an automatic rollback trigger is very different from randomly breaking production. Start with a low stakes, low traffic service before running anything against a critical path.

How is observability different from the monitoring dashboards teams already have?

Traditional monitoring tracks metrics you predicted would matter. Observability is built to answer questions you did not anticipate asking, using traces and high cardinality data rather than a fixed set of predefined dashboards.

Do testers need to learn to write code to work this way?

Reading a trace or interpreting an SLO dashboard does not require deep coding skill, but writing the instrumentation and automating rollback triggers usually does. Most testers moving into this space build up scripting and API skills gradually rather than starting from a deep engineering background.

If reading and testing against real API behavior in production is new territory for your team, the Postman API testing course builds the foundational skills for writing the checks that synthetic monitoring and canary releases both depend on.