How much runs at once

TestVibe ships one runner image that runs in two modes, and a self-hosted install runs one of each:

The split is by network access, not by workload size. Keeping them apart means the interactive path cannot quietly become the execution path.

On Compose they are the runner and browser services; on Kubernetes the -runner and -browser Deployments; on the appliance testvibe-runner.service (port 8081) and testvibe-browser.service (port 8082), both bound to loopback.

Coverage scans get a fifth budget with no setting of its own. They are sized by the same number as concurrent test runs but counted in their own queue, so a full coverage budget can never block a test run, or the reverse.

• They do not compete. Each workload admits against its own budget. A queue of twenty generations never delays a test run, and a saturated load test never stops someone using the assistant.
• First in, first out. Waiters are queued rather than raced, so a job that has waited a while cannot keep losing freed slots to jobs that arrived after it.
• Load is all-or-nothing. A sharded load run waits until its whole set of machines is free, then takes them together. It never starts at reduced scale.
• Your licence can lower one of them. A standard licence caps concurrent test runs at 2 whatever you configure, and excludes load testing entirely. An enterprise licence caps neither. A licence only ever lowers a number — it never raises one you set.

1. The work queues. A run sits in queued / allocating; a generation shows as queued in the app. Nothing has started and nothing is charged.
2. It keeps waiting for up to TV_SANDBOX_ACQUIRE_TIMEOUT_SECONDS — thirty minutes by default. This is why a busy install feels slow rather than broken.
3. Then it is told, honestly. The run finishes with errorClass: capacity_unavailable and retryable: true in its stats — classified as an environment outcome, not a test failure and not a bug. Over the ops API and the CLI the same condition is an HTTP 503 carrying "error": "capacity_unavailable", "creditsCharged": false and a Retry-After: 30 header, so automation can retry without guessing.

How to tell you are at the ceiling

Three signals, cheapest first:

• Runs sit in “queued” for minutes while the box is not obviously busy. That is the queue doing its job, and the cue to raise a budget or add CPU.
• The container log names the pool: [run 1234] FAILED: SandboxPoolBusyException: The 'locked' sandbox pool is at capacity; no session became available in time. The pool name in that message tells you which budget you hit.
• The database tells you exactly. Against your TestVibe database:

-- what currently holds a slot, by workload
SELECT pool, purpose, count(*) AS active
  FROM tv_sandbox_lease
 WHERE status = 'active'
 GROUP BY pool, purpose
 ORDER BY 1, 2;

-- what is waiting behind it
SELECT kind, status, count(*)
  FROM tv_job
 WHERE status IN ('queued','dispatching','running')
 GROUP BY kind, status
 ORDER BY 1, 2;

purpose is run, generation, load, assistant or coverage. An active count sitting at the configured maximum with a non-empty queued row behind it is the definition of “at the ceiling”.

One more limit sits above all four: a self-hosted install allows any single account 25 in-flight jobs of a kind. In practice the four budgets bind long before that does.

Your model endpoint is the other ceiling

A self-hosted install brings its own model, connected on the console's AI providers page. That endpoint has a concurrency limit of its own, and TestVibe does not know what it is:

• TestVibe does not rate-limit your endpoint — the generation budget is what protects it. If your gateway comfortably serves four concurrent conversations, set TV_SANDBOX_GENERATION_MAX=4. Setting it to 10 against a four-way endpoint converts a capacity limit into timeouts and failed generations.
• The model must support tool calling. Generation is a tool loop; a model without it stalls rather than failing cleanly.
• Vision is optional but load-bearing for some features. Tell TestVibe the models accept images only if they actually do — visual comparison and screenshot-grounded healing depend on it.

How long one generation holds a slot

• A generation gets a 90-minute budget by default, adjustable per project between 5 and 120 minutes. Most finish in a small fraction of that; the budget is a ceiling, not a target.
• A generation whose host restarted mid-flight is resumed in place when its browser session is still alive, rather than re-run from the beginning. If it cannot be resumed it is retried at most twice — lower than other work, because re-driving a generation from zero spends real model time.
• So a generation occupies its slot for minutes, not seconds. Three concurrent generations on a small box will keep that box busy for as long as they run.

How concurrent runs are kept apart

Every admitted run takes an exclusive lease, enforced by a unique index in the database, so two runs can never be handed the same slot. What that slot is differs between the hosted service and your install:

Load runs consume capacity differently from everything else. Their budget counts machines, one run can take several at once, and it takes them all or none. Two more settings size a machine:

Virtual users are split as evenly as the shards allow — 90 over 5 machines becomes 18 each — and each machine reports its own metrics, which TestVibe merges into one result. Lowering TV_LOAD_VUS_PER_SANDBOX spreads the same user count over more machines rather than making it cheaper, and lowers the browser-mode ceiling with it.

What happens when a run does not fit

• Above the ceiling — refused before it starts, with the arithmetic. Asking for more users than perMachine × budget (capped at 100 for browser mode) fails immediately rather than being silently reduced. The message spells out the calculation and names the settings to raise: “…requests 120 virtual users but the current capacity is 100 (20 browser VUs/sandbox × 20 load sandboxes, capped at the product limit of 100).”
• Within the ceiling but nothing free — it queues, for the length of its own schedule plus 15 minutes. Admission is all-or-nothing: a 5-machine run waits until all five slots are free and takes them together, holding its place at the front of the queue. Half a load test is worse than none — the numbers look real and are wrong. Smaller runs do not jump it.
• Not licensed at all. Load testing is not included in the standard edition. There a load run is refused before capacity is even considered, with a message naming the edition — so you can tell “not purchased” from “broken”.

Prefer protocol mode for volume. One machine drives up to 1000 protocol users; the same number in browser mode is impossible on any single host. Reach for browser mode when you specifically need real rendering, and keep the user counts modest.

In the admin console

1. Open the console on port 5120 and sign in.
2. Go to Capacity. You should see one field per budget, labelled by workload: Concurrent test runs, Concurrent AI generations, Load-test machines, Concurrent assistant sessions.
3. Change a number and save. It reaches the running app within the settings refresh interval (TV_INSTANCE_CONFIG_REFRESH_SECONDS, 30 seconds by default) — no restart. It applies to the next piece of work that asks for a slot; anything already running is untouched.

In the environment

Docker Compose — add to .env, then recreate the web service:

TV_SANDBOX_LOCKED_MAX=6
TV_SANDBOX_GENERATION_MAX=3
TV_SANDBOX_LOAD_MAX=10
TV_SANDBOX_ROAMING_MAX=6

docker compose up -d web

Kubernetes — set them on the web deployment:

helm upgrade testvibe oci://testvibe.azurecr.io/charts/testvibe \
  --reuse-values \
  --set-string web.extraEnv[0].name=TV_SANDBOX_LOCKED_MAX \
  --set-string web.extraEnv[0].value=6

Appliance — the environment files live on the data disk, so edits survive an upgrade:

sudo nano /etc/testvibe/testvibe.env
sudo systemctl restart testvibe.target

The arithmetic that matters: every concurrent unit of work is a real Chromium browser. Budget roughly 1 vCPU and 1–2 GB of RAM per concurrent slot you actually intend to use, on top of the app, database and object storage. The shipped Kubernetes defaults reflect this — each of the two containers requests 1 vCPU / 2 GiB with a 6 GiB memory limit and a 1 GiB /dev/shm.

Set TV_SANDBOX_ROAMING_MAX (assistant sessions) to about the same number as test runs. These are starting points derived from the per-slot cost above, not benchmark results. Raise one budget at a time and watch two things: runs queueing steadily means you are at the ceiling and should add CPU; runs failing on timeouts means a budget set well above what the box can drive. A queue is a much better failure than a timeout — queued work still succeeds.

Do not size for N concurrent users

Concurrent users is the wrong unit. Most of a user's session is reading results, which costs nothing. Size for concurrent work: decide how many runs, generations and load machines you want in flight at the same moment, give the box a vCPU and 1–2 GB for each, and set the budgets to that number. CI changes the shape more than user count does — a suite occupies exactly one slot for its whole duration, so if your peak is “four pipelines land at once”, size for four.