On the cloud, TestVibe's AI generation runs on a first-party metered service and you never touch a model provider. Self-hosted is different: you decide where the model lives. One of those choices is to point generation at your own OpenAI-compatible endpoint — an internal gateway, an on-prem inference server, or a model running as part of the stack — so generation prompts never leave your network.
This guide covers when that option makes sense, how it's wired, and the caveats that decide whether it actually works.
What the model is (and isn't) used for
Before you stand up any infrastructure, get the scope right. On a self-hosted instance, AI powers exactly two things: test generation and the assistant. Everything else — authoring specs, running Playwright tests, viewing results, load testing — works with no model configured at all. If you never set up a model, generation is simply unavailable and the rest of the product is unaffected.
That scoping matters for capacity planning: you're not sizing an inference cluster for constant traffic, just for bursts of generation work — which is where a GPU earns its keep.
Two ways to supply a model
Self-hosted TestVibe gives you two paths:
- Hosted keys. Provide an Anthropic or OpenAI API key. The app calls that provider directly. Lowest setup effort, strongest quality — but generation prompts leave your network and go to the provider.
- Bring your own model. Point TestVibe at your own OpenAI-compatible endpoint — Ollama, vLLM, LM Studio, or anything that speaks the same API. Nothing leaves your network, which is the whole point for a private or air-gapped install.
The decision usually comes down to one axis:
| Hosted keys | Your own endpoint | |
|---|---|---|
| Setup effort | Lowest — just a key | Higher — run an inference server |
| Privacy | Prompts go to the provider | Stays entirely in your network |
| Quality | Strongest | Depends on the model you run |
| Air-gapped | No | Yes |
| Hardware | None | GPU strongly recommended |
If prompts must never leave your perimeter, the endpoint path is the only one that qualifies.
What you configure
A self-hosted TestVibe is configured entirely from the environment — there is no config file to edit. You set values in the bundle's .env (or inject them as container environment variables), and Compose wires them into the right services. For the custom-endpoint option you supply:
- The endpoint URL of your inference server.
- The model ids it serves. These become the choices in the app's AI Model picker under Settings → AI Model.
- Optionally an API token, a display label, and a multimodal flag if the model handles vision (used for visual comparison features).
Because it's environment-only, any of those values can also come from a mounted secret file instead of sitting in .env: set the file-path form of the variable and TestVibe reads the file's contents. That keeps a gateway token out of your .env and inside a Docker or Kubernetes secret.
The bundle can also run a local model as part of the stack — an optional inference sidecar — so the whole deployment is self-contained without calling out to an external gateway.
The one caveat that breaks generation
Read this before you pick a model. TestVibe does not generate a test by asking for a block of code. It drives a browser step by step, and it does that by having the model call tools. The generation loop is a sequence of function calls — navigate, click, assert, and so on.
That means:
The model must support tool/function calling. A model without it will stall during generation rather than fail cleanly.
Plenty of otherwise-capable local models don't advertise tool support, or implement it unreliably. Choose one that explicitly lists tools in its capabilities — a recent coding-oriented model is your best bet — and verify before you rely on it. A quick check against your endpoint, using the same OpenAI-compatible shape TestVibe will use:
curl http://inference.internal:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "your-model-id",
"messages": [{"role": "user", "content": "Click the submit button on the page."}],
"tools": [{
"type": "function",
"function": {
"name": "click",
"description": "Click an element on the page",
"parameters": {
"type": "object",
"properties": { "selector": { "type": "string" } },
"required": ["selector"]
}
}
}],
"tool_choice": "auto"
}'If the response comes back with a tool_calls array rather than plain prose, the model can drive generation. If it answers in text and ignores the tool, it will stall inside TestVibe the same way.
Two more practical notes: a GPU is strongly recommended — CPU-only inference makes generation painfully slow — and the model's quality directly caps the quality of the generated tests. Hosted frontier models set a high bar; a small local model will produce weaker tests on complex flows. That's a real trade-off, not marketing spin.
Where the safety net still applies
The honest part of TestVibe's generation story survives the model swap. A generated test only reaches generated status after TestVibe replays the whole assembled spec end-to-end and it passes. That validation always runs and can't be turned off. So even a weaker local model can't silently ship a broken test — a test that doesn't actually pass never gets marked generated. What a weaker model costs you is more failed or incomplete generations, not false green checks.
One last thing worth knowing: on-prem, credit gating and token metering are off entirely. There are no credits to buy and nothing is metered. Your only cost controls are your own inference hardware and, if you chose hosted keys instead, your provider's bill.
Running TestVibe fully inside your own network — model included — keeps every generation prompt on infrastructure you control. Get early access to the self-hosted bundle, and see the self-hosted AI models guide for the full configuration reference.