Testing when CI is down

Pick a bad week and you'll find one: GitHub Actions hosted runners backed up in a queue, jobs that normally start in seconds instead waiting, timing out, or never running at all. If your entire test signal lives inside a .github/workflows file, that's the week you shipped blind. Your tests weren't broken. The machine that runs them just never showed up.

That's the failure worth thinking about, and it isn't specific to one provider or one bad day. Runner fleets degrade. Regions have incidents. Outage trackers log a steady drip of CI and platform disruptions across every major provider (ThousandEyes keeps a running timeline). The question isn't whether your CI will have a bad hour. It's what you lose when it does.

If the answer is that all test feedback stops, and you can design your way out of that.

CI is a trigger, not a test runner

Here's the conflation at the root of it. "CI" does two jobs that we've quietly welded into one:

1. It decides when tests should run: on push, on PR, on merge, on a schedule.
2. It provides the machine the tests run on, the hosted runner that checks out your code, installs browsers, and executes.

Those are different concerns. The first is a scheduling problem, the second an execution problem, and when they live in the same system an execution outage takes out your scheduling too. You lose signal for reasons that have nothing to do with either your code or your tests.

Untangle them and the outage changes shape. If the machine that runs the test is independent of the thing that decided to run it, a CI brownout costs you one convenient trigger while every other path to the same test stays open. You degrade gracefully instead of going dark.

Decouple execution from the trigger

The move is to run tests somewhere that isn't your CI runner, and let anything fire them. Push execution into independent cloud sandboxes, a fresh isolated session per run spun up on demand, and the CI job shrinks down to an API call that says "run this suite now, tell me the result." The heavy lifting happens elsewhere. If CI has already dispatched a run, that run can continue independently. If CI never sends the request, use another authorized trigger; no run happens automatically.

This is exactly the seam TestVibe is built around. Tests execute in isolated cloud sessions regardless of who asked. The CI/CD integration is a thin trigger: your pipeline kicks off a run and reads back pass/fail, but if the pipeline is wedged, a previously dispatched run can continue; a blocked dispatch requires another trigger. Same suite, several front doors.

Once execution is decoupled, count how many independent ways you can fire the identical test:

• From CI, on push or PR. This is the default, and the one that just failed you.
• From a schedule, so a nightly or hourly run happens whether or not anyone pushed. A nightly regression playbook that runs on its own clock keeps producing signal straight through a daytime CI incident.
• From another run finishing, via an automation chained to fire when a prior run passes, so a fast smoke pass hands off to the full suite with no pipeline in the loop.
• From a CLI or API call, straight from a laptop or a script. The testvibe CLI, REST ops API, and MCP server all kick off the same run against the same sandboxes. When Actions is down, testvibe run from your terminal is an alternative dispatch path, not a rebuild of your test infrastructure.
• From a click, in the app, when you just want to confirm a thing before you merge by hand.

Five triggers, one execution path. Another trigger can work when its credentials, target and execution dependencies remain available.

The reverse problem: honest gates when the fleet is degraded

Decoupling also fixes the failure that's quieter and more dangerous than "no signal at all": false signal during a degraded window.

When a runner fleet is under stress, jobs don't only fail, they behave weirdly. They get slow. They hit undersized boxes. They time out mid-test. If your deploy gate reads "the CI job didn't come back green," a degraded fleet produces two bad outcomes at once. Real regressions slip through because the job never finished and someone force-merged past a "pending" check. And phantom failures block good code because a test that's fine flaked on an overloaded shared runner, the classic environment drift you get from running on whatever hardware the fleet happened to hand you this hour.

Isolated execution helps on both counts. Isolated sessions reduce shared-state coupling but still depend on available capacity, networking and target health. Inspect the terminal run result rather than equating a dispatch acknowledgment or a pending CI job with a pass.

Two practical rules fall out of this:

• Make the job exit reflect the terminal test result. A CI job can be yellow for infrastructure reasons that say nothing about your code. Read the actual run outcome from the system that executed it.
• Keep at least one trigger that doesn't share fate with your primary CI. A schedule or a run-chained automation on a separate path means an Actions incident degrades convenience, not confidence.

The test for your setup

Ask one question: if my CI provider had a two-hour outage right now, would I still have test signal?

If the honest answer is no, your tests are coupled to your CI runner, and you're one bad status-page morning away from shipping blind. A second independent CI path can also help if it can build/deploy and dispatch the intended target, and neither will a pile of retry logic. Move execution off the runner entirely, and the runner becomes one trigger among several: still the convenient one, no longer the thing everything hangs on.

Check whether, during the next Actions incident, you can type testvibe run from your terminal and dispatch the intended tests, wait within a bounded budget and obtain an honest terminal outcome. If you can, the outage cost you a notification. If you can't, it cost you the whole afternoon.

Want tests that run in isolated cloud sessions you can trigger from CI, a schedule, a chained run, the CLI, or a click? Get early access, or read how the CI/CD integration wires a run into your pipeline as a thin trigger.