Why regression testing is critical today

What regression testing actually is

Regression testing re-verifies behaviour that already worked, after a change. It is the opposite of testing a new feature: nobody asked for the checkout to change, so if it behaves differently after today's release, that difference is the finding.

It is also not the same as retesting. Retesting confirms one specific bug is fixed. Regression testing asks the broader question the fix raises — what else did that change touch? A single-line fix to a shipping rule can move a total, a template edit can drop a form field, and a dependency bump can change how a date is parsed.

The trigger is any change, not just your own code: a plugin or package update, a config or environment change, a CDN or DNS switch, a third-party script updating itself, content edited in a CMS. Each one can alter behaviour you never touched, which is why regression testing is a continuous habit rather than a release-day event.

Why regression testing matters more now

Generative AI changed the economics of writing code. A feature that took a day takes an hour; a refactor nobody had time for happens on a whim. What did not change is the need for the result to be correct — and the failure mode of an AI assistant is unlike a human's. It does not stall, hesitate or leave a TODO. It produces something plausible, complete and fluent, whether or not it is right.

That shows up in concrete ways. Models invent things that don't exist: a study of package hallucinations found that roughly one in five packages recommended by code-generating models were not real, which is both a bug and a supply-chain risk, since an attacker can register the hallucinated name. They also rewrite more than asked — GitClear's analysis of hundreds of millions of changed lines reports rising duplication and falling refactoring, the signature of code being added rather than reorganised.

The perception gap is the dangerous part. In a randomised trial by METR, experienced open-source developers working on their own repositories were measurably slower with AI assistance — while believing they had been substantially faster. Feeling fast is exactly when verification gets skipped, and Google's DORA research has linked rising AI adoption with reduced delivery stability. Speed without a safety net is how a quiet regression reaches customers.

There is a second-order effect, too: more of the code in your repository is code nobody on the team wrote from scratch, and reviewed in larger diffs than anyone reads carefully. Institutional memory — 'careful, that function is load-bearing' — thins out. Automated regression testing is what replaces the memory.

The types of regression testing

Regression testing is not one technique. The classic types describe how much you re-run: unit regression re-runs the tests around the changed unit only, in isolation. Partial (selective) regression re-runs the changed unit plus what depends on it, chosen by impact analysis — the usual default, because it is fast enough to run on every commit. Complete regression re-runs everything, reserved for a large or risky change such as a framework upgrade, a platform migration or a long-lived branch finally merging. Retest-all is the brute-force version: run the entire suite, change or no change, usually on a schedule rather than per commit.

Two more describe what kind of change you are guarding: corrective regression re-runs existing tests unchanged, because the code changed but the specification didn't. Progressive regression updates the tests first, because the specification itself moved — new behaviour is expected, and the point is to confirm that nothing outside the new behaviour moved with it.

Then there are the layers, which matter more than the labels. Functional and end-to-end regression drives real user journeys — add to cart, check out, pay, receive the confirmation — and is the only layer that proves the business still works. API and contract regression pins the shape of a response so a field that quietly changes type is caught before consumers break. Visual regression compares rendered pixels against an approved baseline, catching what a DOM assertion never will: a layout collapsing, a font failing, an image vanishing. Structure drift works one layer down on the DOM, catching what pixels miss, such as a tracking script or form field that disappeared without changing the picture. Performance regression compares timings against a baseline, because a page that still works but now takes four seconds is a real regression. Data and schema regression re-checks migrations and reports against known inputs. Accessibility and SEO regression catch the invisible-to-you breaks: a lost label, a noindex rule that shipped with a staging config.

Where your test suite stops

Every type above tests the code you control, at the moment you change it. Your live site is assembled from much more than that: plugin and package auto-updates, a theme update, a payment gateway changing its embed, a third-party script updating itself overnight, a CDN rule, a PHP or Node version bump by your host, content edited by someone in a hurry.

None of those touch your repository, so none of them run your test suite — and all of them have broken production sites. The most common failure we see is not a bad deploy at all: it is a WordPress or WooCommerce plugin updating itself at 3 a.m. and changing the checkout, on a site whose code did not change that week. A green CI pipeline is silent about every one of these.

That is the gap continuous production checks fill: the same regression testing discipline, pointed at the running site rather than the build. The baseline is what your site did yesterday; the finding is what changed today, whoever changed it — including the people and robots who never opened your repository.

What to watch continuously, in order

Start with the money path. One end-to-end check of the journey that pays the bills — add to cart, checkout, order placed, confirmation email received — is worth more than a hundred unit tests, because it is the one failure that costs money per hour. Relvato runs it as a checkout monitoring journey against the real store, on a schedule and after every update.

Then the two rendering layers, which catch different halves of a broken deploy: visual regression for what the page looks like, and structure drift for how it is built. Add Core Web Vitals so a slow regression is treated as the regression it is, and an exposure scan so a key or backup that appears in a build is found the day it appears rather than by someone else.

The ordering principle is simple: protect what breaks loudest first, then widen. A small set of checks that actually run beats a comprehensive suite nobody maintains — and unlike a test suite, production checks keep their value even when the change came from outside your repository.

If you have no test suite at all

Plenty of sites that make real money have no automated tests, and the advice to 'write a test suite first' is how that stays true for another year. There is a shortcut: start at the outside. A handful of production journeys can be running today without touching your codebase, and they will tell you within minutes of a bad change that something a customer relies on no longer works.

From there, push downward. When a production check catches a regression, that is the behaviour worth pinning with a unit or integration test, written with the failure in front of you. Your suite then grows from real incidents rather than guesses about what might break — which is also the cheapest way to build one.

The uncomfortable truth of the AI era is that generating code is no longer the bottleneck; knowing whether it works is. Regression testing is how you find out on purpose, instead of finding out from a customer.