Before letting an AI tool explore your app, ask what leaves your network and where it is processed. DOM content, screenshots, URLs, and flow descriptions may contain sensitive information. Map the tool’s data path to your security and procurement requirements before choosing a deployment.
This is the part of "AI-generated tests" that engineers wave through and their compliance team does not. So let's be precise about what an AI test generator actually sends off-box, then map the honest spectrum of where you can make it run.
What actually leaves when a model writes a test
To generate a real test, the model can't work from your one-sentence description alone. It has to see the app. In practice that means some combination of:
- The URL under test — often a staging or internal hostname that itself reveals infrastructure.
- The rendered DOM — your markup, element labels, form fields, and the copy on the page.
- Screenshots — literal pixels of the page, including whatever data is on screen at that moment.
- Your description of the flow — which frequently encodes business logic ("an admin approves a wire transfer over the daily limit").
- The generated code and its run artifacts — traces, videos, logs.
None of that is exotic — it's the necessary raw material for grounding a test in your real UI instead of guessing at selectors. A model has to explore your live app before it can write a line of code. But "necessary" and "allowed to leave the building" are different questions, and the second one is answered by your data-residency posture, not your enthusiasm for AI.
The reassuring news is that where this material goes is a choice, not a fixed property of the tool. The tradeoff is operational weight against perimeter control, and it lives on a three-point spectrum.
Point one: cloud, first-party metered AI
The default. You sign in, describe a feature, and generation runs on TestVibe's cloud. The AI is a first-party metered service — you do not bring a model-provider key. Hosted generation still uses external infrastructure and model providers, so review the service's processing terms and subprocessors for your residency requirements. Generation runs against your app inside an isolated per-run sandbox: a fresh browser context that's created for that run and torn down after, so one run's state never bleeds into the next.
Credentials are the sharp edge here, and they're handled deliberately. Login passwords and API tokens live in a per-project encrypted store and are referenced from your spec as tokens like {{secret:TEST_USER_PASSWORD}}, never as literal values. Values are injected into execution environments, including generation sessions that must exercise login. Generated source should refer to secret names rather than embed values, and transcripts redact known secrets. A login can still appear in browser evidence; review traces and screenshots before sharing them.
For most teams — public apps, ordinary staging environments, no residency clause — this is the right answer, and worrying about model geography is worrying about the wrong thing. The cloud is genuinely the least to operate.
Point two: private runner (a cloud feature, not self-hosting)
Before jumping to a full on-prem install, know the middle option, because it addresses a narrower requirement. If your only constraint is the app under test must not be reachable from the public internet — an internal console, a VPN-only staging box, a build on your laptop — you don't need to self-host anything. You run the hosted app as normal and route execution through a tunnel or a private runner inside your firewall. The app URL stays unchanged, no inbound ports open, while a tunnel carries browser traffic between the cloud execution session and your private app. A private runner places execution in your network, but hosted control-plane and artifact traffic still need a separate review.
This is a data-reachability answer, not a full data-residency one: with a tunnel, generation still happens in TestVibe's cloud. Reach for it when the gate is "our app can't be public," and reach past it when the gate is "the control plane itself can't be hosted."
Point three: self-hosted, and self-hosted with your own model
When the requirement is that the whole platform runs on your infrastructure — air-gapped deployments, strict residency rules, a compliance regime that won't allow a hosted control plane — the self-hosted edition puts every service inside your network. It's a Docker bundle: the app, Postgres holding your projects and test files, an object store for artifacts, the runner, and a browser host. Your DOM, screenshots, traces, and canonical test files live in volumes you own. Billing and metering are off entirely; there are no credits and nothing is counted.
That still leaves one question: where does the model run? Self-hosted gives you two answers. Supply a hosted Anthropic or OpenAI key for a hosted-provider setup — at the cost of generation prompts leaving your network for that provider. Or point generation at your own OpenAI-compatible endpoint — an internal gateway, a vLLM or Ollama server, or an inference sidecar baked into the bundle — and generation can stay internal when the target and every enabled integration are also internal. For an air-gapped install, the endpoint path is the only one that qualifies.
There's one hard requirement on that model: it must support tool/function calling, because generation drives a browser by having the model call navigate, click, and assert step by step. A model that can't call tools stalls rather than fails cleanly. Plan on a GPU, and know that the model's quality caps the quality of the tests it produces.
A capable open-weight model can make an internal endpoint practical, but a leaderboard result does not establish quality on your app. Check distribution rights, hardware requirements, tool calling, integration egress and the actual generation result for the deployment you plan to use.
The safety net doesn't move
Wherever the model runs, one guarantee holds: a generated test only reaches generated status after TestVibe replays the whole assembled spec end-to-end against your live app and it passes. This validation can't be turned off. A weaker or private model may need more retries. Passing verification still does not prove that its assertions express the right requirement; review the spec. And once a spec is generated and verified, it's deterministic Playwright: ordinary Playwright runs do not require a generation model. Optional AI assertion plugins or AI triage can still make model calls. The authored assertions stay in the spec until you edit or regenerate it; app state and integrations can still affect results. Review runtime integrations as well as generation when assessing residency.
The buyer's checklist
Strip the marketing off any AI testing tool and ask four questions:
- What leaves? DOM, screenshots, URLs, flow descriptions, generated code — assume all of it during generation unless the vendor says otherwise.
- Where is it stored, and for how long? Artifacts and test files: whose disk, whose region, what retention?
- Who can read it? Review the hosted service and its subprocessors, or the providers and integrations you configure on a self-hosted installation.
- What's the private-perimeter path? Tunnel for reachability, self-hosted for the control plane, custom endpoint for prompts that never leave — and which one your regime actually requires.
The right answer isn't "always on-prem." It's matching the point on the spectrum to the gate you actually have to clear, and knowing that with open weights at frontier level, model quality and operational cost still need to be measured on your own workloads.
See where your data would go
Map your own requirement before you buy: reachability, residency, or full air-gap. Get early access to try the cloud edition, or read the self-hosting guide and the custom-endpoint walkthrough to map which services and integrations stay inside your network.