Turn a production incident into a regression check

An incident is fixed and the postmortem says "add a regression test." The useful next step is concrete: confirm the failing journey, write the expected result, and assign the check an owner and a run schedule. This guide follows that path using a checkout example.

A fix and a regression check answer different questions. The fix addresses the current defect; the check records an expectation that future changes can exercise. Keep the reproduction and expected outcome with the test so someone else can understand why it exists.

Reproducing the state, selecting locators, preparing data, and reviewing assertions all take work. Test generation can help with authoring, while reproduction and review remain explicit steps.

Reduce the authoring friction while keeping reproduction and review explicit. Here's the loop that actually closes.

1. Capture the available error evidence

You can't write a regression test for a failure you never saw. Instrument the environment that produces the error. TestVibe’s .NET and Node.js agents report server/process metrics and supported application errors; they do not automatically capture browser JavaScript exceptions, user session replay or field web-vitals history. Add appropriate browser instrumentation separately when needed. Inspect the captured stack and metadata, then corroborate the reproduction.

2. Reproduce it from what was captured, not from a guess

Now you reproduce. The trap here is debugging from imagination: reading a stack trace and guessing at the sequence of clicks that produced it. Guessing is slow and often wrong, and a regression test built on a wrong reproduction protects against a bug that never existed.

A production error doesn't hand you a recorded session. It can supply grouped errors and available metadata; a URL or complete reproduction is not guaranteed. Follow the same route, hit the same edge case, and confirm you're standing up the same exception TestVibe grouped, not a lookalike that happens to throw from the same function.

By the end of this step you have a precise, confirmed answer to the only question that matters for a regression test: what sequence of user actions produces this failure?

3. Describe the failing behavior in plain language

Use the confirmed reproduction as the starting point for a behavior description.

You already know the reproduction. Instead of hand-writing the Playwright (importing fixtures, picking resilient locators, wiring the assertion), you describe the behavior the bug violated:

Given a user with an empty cart
When they apply a promo code and then remove the last item
Then the cart total shows $0.00 and no negative price is displayed

The example records the expected cart behavior. TestVibe generates Playwright code and only reports generated status after the assembled spec passes verification against the app. Review the assertions and setup to confirm that they exercise the intended regression. A passing run establishes the checked outcome in that environment.

Generation can reduce manual authoring, but review, data setup, and debugging still take work. Confirm that the assertions express the intended behavior rather than merely producing a passing run.

4. Schedule the regression check

A regression test that runs once is a fossil. The whole value is that it runs after you've forgotten the incident, on the exact change that would reintroduce the bug.

Choose when the check should run: on relevant changes in CI, on a schedule, or both. Scheduled automations can rerun the suite nightly or after another run finishes, and Failures notifications can route failed outcomes when configured. Inspect the Playwright trace when retained. A scheduled check detects what it exercises when it runs; it does not guarantee detection before a user encounters the defect.

That's the loop, closed:

Production error captured → reproduced from what was captured → described in plain language → generated and verified → scheduled re-runs.

Why this changes the economics

Each step makes the action item easier to complete: telemetry supplies available evidence, reproduction confirms the path, the description records the expectation, and scheduled runs keep exercising it. Missing context still needs investigation; one sentence does not replace setup and review.

Keep the incident’s reproduction, expectation, and test together. That makes the check easier to review and maintain after the original responders move on to other work.

For the negative-total example, save the cart setup and the $0.00 expectation with the check. Run it against the fixed build, review its assertions, and include it in the relevant CI gate and scheduled suite. The next refactor then has a specific behavior to verify.

Want to see the loop end to end — telemetry capture, plain-language generation, scheduled runs with retained diagnostic evidence? Get early access, or read how generation works.