Self-hosted · Configuration

AI providers

What TestVibe uses a model for, how to connect one on your own infrastructure, and what it costs when the AI is yours. Sign in with the email on your subscription.

We send a 6-digit code. Access is limited to emails on an active self-hosted subscription — talk to sales if you do not have one yet.

We sent a code to . It expires in 10 minutes.

Self-hosted · Configuration

AI providers

On the cloud, AI is a first-party metered service and you never see a key. Self-hosted is the opposite: you bring the models, you pay the provider directly, and TestVibe meters nothing. This page is how to connect one, which one gets used, and what happens when it is missing.

01

What AI is used for

Not everything in TestVibe needs a model. Knowing which parts do tells you what degrades when a provider is missing — and it is less than people expect.

FeatureNeeds a model?Where the call is made
Generating tests from a plain-English feature, and self-healing a broken one Yes An agent program running in the generation sandbox, reaching the model through the application
The in-app assistant — chat, exploration, headless assistant automations, AI triage of a failed run Yes In-process, from the web service
Visual comparison — "does this page still look right" Yes, and it must accept images In-process, from the web service
Running existing tests, results, traces, reports No —
Load testing No —
Sign-in, projects, scheduling, notifications, connectors, hooks No —
An install with no AI configured is still a working install. You can import or hand-write Playwright specs, run them on a schedule, load-test, and read every result. What you lose is the part that writes tests for you.
02

Connect a provider

You tell TestVibe which AI company — or which machine of your own — to call, and which of its models your team may pick. It is all done in the console, and none of it needs a restart.

The five kinds

Every provider has one kind, which is how TestVibe knows what language to speak to it. The list is closed — these five, no others.

KindWhat it isYou supply
anthropic Claude models, direct from Anthropica key
openai GPT models, direct from OpenAIa key
openrouter OpenRouter — hundreds of models, one keya key
opencode OpenCode Zen — a curated set, one keya key
custom Any other endpoint speaking the OpenAI chat-completions API — Ollama, vLLM, LM Studio, TGI, LocalAI, LiteLLM, Azure OpenAI. This is how you use a model inside your own network. a name and a base URL; key and model ids optional

The first four are offered by name, so you only paste a key. Anything else is Custom.

Do it

  1. Open the console on port 5120 — http://<your-host>:5120/, or /admin on the appliance. Go to Setup → AI models. Success: you see a Connected list and, below it, Available.
  2. Connect a named company. Press Connect, paste the key, press Connect again. Leave the box empty if this server already carries that key in its environment. Success: it moves into Connected.
  3. Or describe your own endpoint. On the Custom row press Add new. Name and base URL are required; key and model ids are optional. Give the root only, e.g. http://ollama:11434 — TestVibe posts to {base URL}/v1/chat/completions and reads the model list from {base URL}/v1/models. Success: a card appears under Connected.
  4. Switch it on and tick its models. Press Get latest list, tick what your team may choose, then turn on the switch at the top of the card. Success: the company is on and a model is ticked.
  5. Prove it in the app: a project → Settings → AI Model. Success: your model is listed under the provider's name.
The mistake that looks like a broken install A provider you just added arrives switched OFF, with no models ticked. That is deliberate — adding an endpoint and offering it to users are two decisions — but stop after pasting the key and your users see no change at all. Both switches must be on: the company, and each model.

The key is encrypted, stored in this install's own database, and never shown again.

Two checks on your own endpoint

The important one: tool calling Pick a model with reliable OpenAI-style tool calling. The agent drives everything through tool calls. A model that emits malformed tool_calls, or emits them only sometimes, does not fail cleanly — it stalls and times out. Known-good: Qwen2.5-Coder or Llama-3.1-Instruct on vLLM, or an Ollama model whose card advertises tools. Check from the TestVibe host, not your laptop:
does it answer at all?
curl -fsS http://ollama:11434/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"qwen2.5-coder:32b","messages":[{"role":"user","content":"say ok"}]}'
Visual comparison degrades, it does not break When the chosen model came from any OpenAI-compatible provider — Custom, OpenRouter or OpenCode — TestVibe sends screenshots only if TV_LLM_VISION is set on the web service. Otherwise you get a plain explanation instead of a comparison: "Visual comparison is not available on the configured self-hosted model." Awkwardly, that switch is an environment variable even for a provider you connected in the console.
03

Which engine runs a generation

Generating a test is the one job not called from the web service: it starts a coding agent inside a sandbox. There are three such agents, and the provider's kind — not the model's name — decides which one runs.

Provider kindEngineKey it uses
anthropic Claude ANTHROPIC_API_KEY in the web service's environment, and only there — see section 04.
openai Codex That provider's own stored key; failing that, OPENAI_API_KEY.
openrouter
custom		 OpenCode That provider's own stored key; failing that, OPENROUTER_API_KEY or OPENAI_API_KEY.
a model no connected provider carries Claude ANTHROPIC_API_KEY — the safe fallback, and what an install that connected nothing gets.

Only a provider registered on the AI models page can move a generation off the Claude engine: that is the one place TestVibe holds the kind, the endpoint and the key together.

OpenCode Zen is left out of this table on purpose Connect the OpenCode provider for the assistant if you like, but ask support@testvibe.com before planning generation around it. The four kinds above are the ones we can tell you work.
04

Generating tests with no internet

The question every disconnected site asks first. Yes, this works — so long as you connect your endpoint on the AI models page.

A Custom provider registered there runs generation on the OpenCode engine with that provider's base URL and that provider's own key. Nothing in the path touches Anthropic, and no ANTHROPIC_API_KEY is needed or looked at.

With the sandbox's outbound lockdown on, a run opens only the model host it needs — your provider's base URL for a Codex or OpenCode run, api.anthropic.com for a Claude one. The site under test is a separate allowance, and it is the one people forget on a segmented network.

A model that exists only in TV_LLM_MODELS still needs Anthropic The older environment endpoint (section 08) adds models to the picker but registers no provider. Generation therefore falls back to the Claude engine and refuses to start without ANTHROPIC_API_KEY — which a disconnected box cannot use anyway. Register the endpoint on the AI models page as a Custom provider instead.
Connecting Anthropic in the console does not enable generation An Anthropic key entered in the console powers chat, the assistant and visual comparison. It does not power generation: the Claude engine reads ANTHROPIC_API_KEY from the web service's environment and nowhere else.
05

Which model gets used

Choosing a model is a per-project setting your users make. What you control is which models exist to choose from.

The picker is Settings → AI Model in a project. Its list is built from three sources: the models that ship with TestVibe (claude-opus-5, the default, and claude-fable-5); every ticked model of every switched-on provider you connected; and every id in TV_LLM_MODELS. They are shown grouped under whoever makes them, friendly name first, raw id underneath.

If no hosted key is set anywhere, the default moves to the first model that can actually run, so a project is never pointed at something this install cannot reach.

A project can also pick a different model per job. There are four: writing tests (this is the one that also picks the engine — see section 03), the assistant, analysis (including visual comparison), and quick helpers, which default to claude-haiku-4-5 when this install can reach it. A project that chooses nothing per job uses its one project-wide model — which is what every project does until someone changes it.
06

Missing or unreachable

What your users actually see, so you can recognise each case from a support message rather than from a log dive.

SituationWhat happensWhere to look
Provider connected, picker unchanged Users see exactly what they saw before. Nothing fails, which is why it is hard to spot. The provider is off, or no model is ticked. Section 02. The most common report by far.
Nothing configured at all Generation fails at once with "The generation service isn't fully configured for this engine yet — please retry later." Everything else works. Web service log: engine=claude-code: no ANTHROPIC_API_KEY configured on the web host — cannot run
A connected provider has no key, or a Custom one no base URL Generation fails naming the provider — has no API key configured or has no endpoint URL configured. Fix it on that provider's card in the console.
Your endpoint is configured but down Assistant replies and Gherkin help fail with the transport error. Runs, schedules and reports are unaffected. Web service log, then curl the endpoint from the host.
Endpoint up, model has weak tool calling Generation makes no progress and times out — the worst failure mode, because it looks like slowness. Change the model. Section 02.
A model id does not match the endpoint Model-not-found on first use. Ids match exactly, case and any :tag included. Get latest list copies them for you.
Key valid but out of credit or rate-limited The provider's own error surfaces on the generation. TestVibe adds no gate of its own. Your account with that provider — TestVibe cannot see your balance.
Not a capacity problem A generation that queues rather than fails is an admission-cap question, not a provider question. See Capacity.
07

Cost and metering

On self-hosted, TestVibe does not meter, does not charge and does not gate. Whatever the model costs is between you and your provider.

What is switched off

  • No credit gate on runs, generations or load tests — the pre-flight always allows.
  • No token metering, no ledger debit, no usage row for an AI call — so expect no in-app cost report. The provider's own dashboard is the authority on tokens and spend.
  • No credit reservations, no kill-on-zero watchdog and no "buy credits" prompt anywhere.
  • Stripe is dead configuration here. The appliance drops TESTVIBE_STRIPE_SECRET_KEY altogether; the Compose deployment still carries it, blank, and nothing reads it.
The consequence worth planning for There is no spend ceiling inside TestVibe. A user who regenerates a large suite repeatedly will spend real money on your provider account and nothing in the product will stop them. Set the limit where it can be enforced — a spend cap and budget alerts on the provider account — and use the concurrency caps to bound how fast it can happen. A model on your own hardware has no per-token cost at all; the trade is that you supply the GPU, and quality depends entirely on the model you chose.
08

Appendix: the older TV_LLM_* settings

Before the AI models page existed, one endpoint was described with environment variables. Every name still works, so an install that sets them keeps running untouched — but none appears in any screen any more, and they cannot register a provider. Use section 02 for anything new.

VariableWhat it does
TV_LLM_BASE_URL Endpoint root, e.g. http://ollama:11434. A trailing /v1 is tolerated. Required.
TV_LLM_MODELS Comma-separated model ids, exactly as the endpoint names them. At least one required.
TV_LLM_API_KEY Optional bearer token, sent only when set. Ollama needs none.
TV_LLM_LABEL Name shown beside those models in the picker. Defaults to Local.
TV_LLM_VISION Set to 1 only if the model accepts images. Also gates visual comparison for console-connected OpenAI-compatible providers.
TV_LLM_SEND_TEMPERATURE Send a temperature parameter. Off by default — some local servers return 400 on parameters they do not know. vLLM accepts it.

Those models appear only when the install is self-hosted and TV_LLM_BASE_URL is set and TV_LLM_MODELS lists at least one id; a half-configured group silently offers nothing, and no screen tells you so any more. Every TV_LLM_* name supports the mounted-secret form {NAME}_FILE. ANTHROPIC_API_KEY, OPENAI_API_KEY and OPENROUTER_API_KEY do not — they are plain environment variables, so changing one means restarting the service. A key entered in the console is the opposite: no file, no restart. Anything pinned in the environment beats anything stored in the database and shows as "locked by environment". Both shipped deployments already set TV_SELF_HOSTED=1.

Where to go next