← All posts
Concepts

p95 is a promise: reading load-test results like an SRE

A load test spits out a wall of numbers, and the temptation is to grab the average and move on. Averages lie. The metrics that actually predict how your app behaves under real traffic are percentiles, and reading them well is the difference between a green run and a green run that ships an outage. This is how to read a load-test result the way an SRE does.

Why the average hides your worst users

Say you run a checkout journey with 200 requests and record these latencies. Most land near 300 ms, but ten of them take 4 seconds because a connection pool is saturated. The math:

mean          = 485 ms   ← looks fine
p50 (median)  = 300 ms   ← the typical user
p95           = 900 ms   ← 1 in 20 users
p99           = 3800 ms  ← 1 in 100 users

The average tells you 485 ms and everyone nods. But 1% of your users — the p99 — are waiting nearly 4 seconds, and at real traffic that 1% is thousands of people per hour. A single mean can't show you that tail because the fast requests drown it out.

Percentiles are ordered promises. p95 = 900 ms means "95% of requests finished in under 900 ms." It's a commitment you can put in an SLO and gate a deploy on. The average is a description of nothing in particular.

Rules of thumb worth internalizing:

Ramp, don't slam

How you apply load matters as much as how much. Dumping 500 virtual users on a cold system in one second measures your cold-start behavior, not your steady-state capacity. Both are worth knowing, but don't confuse them.

A realistic profile ramps virtual users from zero to the target over a warm-up window, then holds:

// k6 stages: ramp to 100 VUs, hold, ramp down
export const options = {
  stages: [
    { duration: '2m', target: 100 },  // ramp-up: 0 → 100 VUs
    { duration: '10m', target: 100 }, // steady state: hold 100 VUs
    { duration: '1m', target: 0 },    // ramp-down
  ],
};

Keep the ramp at roughly 10–20% of the total duration so connection pools, JITs, and caches warm the way they would in production. Then read the hold phase for your capacity numbers — the ramp phase is noise for that question. A common mistake is reading the peak-latency spike during ramp-up and declaring the system broken, when it just needed thirty seconds to warm up.

Watch two curves together: active users and throughput. If VUs climb but requests-per-second flatlines, you've found your ceiling — adding load isn't producing more work, it's just producing more waiting.

Spend an error budget, don't blow it

"Zero errors" is the wrong target for a load test. At the edge of capacity, a tiny, bounded error rate is acceptable — that's what an error budget is for. Pick a threshold and hold the run to it:

export const options = {
  thresholds: {
    http_req_failed: ['rate<0.01'],    // allow up to 1% failures
    http_req_duration: ['p(95)<1000'], // 95% under 1 second
  },
};

A run passes only if every gate holds for the whole run. That last part matters: a threshold that's true on average but violated for a 90-second window still failed, because that window is a real outage your users lived through.

When errors do appear, correlate them with the load curve. Errors that start exactly when VUs cross a certain number point at a hard limit — a connection pool, a rate limiter, a thread cap. Errors scattered randomly at low load point at something flaky in the app itself. The shape of the error-over-time curve tells you which.

Spotting saturation before it spots you

Saturation is when the system runs out of some resource and latency starts climbing while throughput stops. The earliest, cleanest signal is a rising p95 under steady load. If your VU count is flat but p95 drifts upward over the hold phase, something is filling up — a queue, a pool, memory, disk. You're watching a leak or a backlog form in real time.

The classic saturation signature, all at once:

When you see the percentile lines diverge like that, you've found your knee — the load level where the system stops scaling linearly. That number is your real capacity, and you want a comfortable margin below it in production.

Where TestVibe fits

TestVibe's load testing gives you exactly these views without wiring up your own tooling. Simple mode runs concurrent browser users that replay your generated feature tests and time every step, so you get per-step p50/p95/p99 and can see which step of the journey degrades first. Advanced mode generates a protocol-level k6 script with virtual users, a ramp-up window, and threshold gates like p(95)<1000 or rate<0.01 — the run fails if any gate fails, for the whole run.

A load-test result showing p50/p90/p95/p99 latency over time, a p95 KPI tile, failed-run count, and peak concurrent users.
A load-test result showing p50/p90/p95/p99 latency over time, a p95 KPI tile, failed-run count, and peak concurrent users.

The results view puts the percentile-over-time chart, the throughput and active-users curves, and the error rate side by side — which is precisely the correlation you need to tell warm-up from saturation. Set your thresholds from a passing baseline run rather than guessing, then tighten them as results stabilize.

Read percentiles, not averages; read the hold phase, not the ramp; and gate on a budget you can defend. Do that and a load test stops being a wall of numbers and starts being a promise you can keep. Get early access to try it, or read Run a load test and read results for the full walkthrough.

Early access

Ready for tests that write themselves?