From plain English to Playwright: how TestVibe generates tests that actually pass

Every AI code tool can produce a Playwright file that looks right. The hard part is producing one that passes — on the first run, in a fresh browser, against your real application, and again next week when nobody is watching.

That gap is the whole design problem behind TestVibe's generation pipeline. This post walks through what actually happens between "I typed a sentence describing my feature" and "a green run," from the user's side of the glass: how your description becomes a Gherkin spec, why the AI agent explores your live site before writing a line of code, what the generated code looks like, and what happens when your app inevitably changes.

Step 1: Describe the behavior, not the implementation

Everything starts in the Features view of your project. You pick a group, choose the action to create or generate a feature, and describe the behavior you want covered — in plain English.

The quality bar here is lower than you might fear, but not zero. Compare:

Test admin stuff.

versus:

Create a feature for an admin inviting a new team member and seeing a success message.

The second one names the actor, the action, and — critically — the visible outcome that proves success. That last part matters most, because a test that clicks around without asserting anything is theater. TestVibe's guidance is the same formula testers have used forever: Given a starting context, When the user acts, Then something visible proves it worked.

You don't have to start from a blank text box either. The project Assistant can explore your app and suggest tests, propose what to test first, point out coverage gaps, or generate a test straight from a description you type into chat.

Step 2: The Gherkin draft — a spec you can actually read

Your description becomes a .feature file in Gherkin: a structured, human-readable spec with a Background (shared setup), Scenarios, and steps. This is the contract between you and the generated code — if the Gherkin is wrong, the code will be wrong too, so TestVibe opens the draft for review and editing before any Playwright code exists.

A well-drafted feature has a few properties worth knowing about:

• It starts at the entrypoint. The Background reaches the starting screen the way a real user would — landing page, sign-in, navigation — rather than assuming an already-logged-in, already-navigated state. A test that "assumes" its starting point works on your machine and nowhere else.
• Secrets are tokens, not values. Credentials come from Settings → Variables & Secrets and appear in the spec as tokens like @LOGIN_USERNAME, so passwords never live in test text.
• Created data is unique per run. Anything the test creates gets wrapped in {{unique}} (e.g. a project named "Project {{unique}}"), so a repeat run doesn't collide with data the last run left behind.
• Steps name visible things. When I click the "Add to cart" button on the "Sauce Labs Backpack" product card gives the generator (and the next human reader) something concrete to find.

Here's the full arc for a small example. You type:

Cover removing an item from the cart and verifying the cart badge disappears.

TestVibe drafts something like:

Feature: Cart removal
  Background:
    Given I am on the "Login" page
    And I sign in with username @LOGIN_USERNAME and password @LOGIN_PASSWORD
    And I click the "Add to cart" button on the "Sauce Labs Backpack" product card

  Scenario: Remove an item from the cart page
    When I click the shopping cart icon in the header
    And I click the "Remove" button next to "Sauce Labs Backpack"
    Then the cart list is empty
    And the cart badge in the top-right is not shown

You review it, tighten a step if needed, save — and only then does code generation start.

Step 3: Live exploration — the agent uses your real app first

This is the part that separates generation from autocomplete. When you start generation, TestVibe spins up an isolated cloud test session, and the AI agent opens your actual site and works through the same phases every time:

1. Exploring the app. The agent probes the UI the feature describes — finds the real buttons, the real labels, the real timing quirks — before committing to any code.
2. Writing the test. It authors the Playwright code for each scenario, grounded in what it just observed.
3. Verifying, run N. It replays what it wrote against the live site. A failed verification loops back to writing and tries again.
4. Final verification. Once every scenario is recorded, TestVibe runs the whole assembled spec end-to-end.

You watch all of this live in the generation drawer: section-by-section progress through Background and each Scenario, per-step credit/time/tool-call counts, screenshots of what the browser saw, and trace links. If the agent hits something genuinely ambiguous, generation can pause and ask you a question mid-run, which you answer right in the drawer.

Exploration is also why generated tests don't guess at selectors. The agent isn't pattern-matching on what login forms usually look like — it found your login form, typed into it, and watched what happened.

Step 4: Code you'd accept in review

The output is a standard Playwright test file, saved into your project alongside the feature it belongs to. Two quality properties are worth calling out.

Web-first assertions. Generated code proves outcomes with Playwright's auto-retrying expect matchers on locators — toBeVisible, toHaveText, toHaveCount — rather than grabbing a value and comparing booleans. The difference is the whole flake-vs-reliable divide:

// What TestVibe generates: the matcher waits and retries
await expect(page.getByText('Sauce Labs Backpack')).toBeHidden();
await expect(cartBadge).toHaveCount(0);

// What you won't find: a race condition wearing an assertion costume
const visible = await badge.isVisible();   // snapshots one instant
expect(visible).toBe(false);               // passes or fails on timing luck

Readable locators. Roles, names, and visible labels over brittle generated IDs — page.getByRole('button', { name: 'Login' }) — so the code survives small layout changes and a human can tell what it does.

Step 5: Verified before it's delivered

A feature only reaches generated status after final verification passes — the complete assembled spec, run end-to-end. There is no way to skip this check, and that's deliberate. Recording each step successfully is not the same thing as the finished test passing a real run.

The reason is a principle that shapes the entire pipeline: a test must be proven under the same conditions it will actually run in. The agent works in a warm, already-navigated session at human speed; your scheduled test runs cold — fresh browser, fresh context, from the entrypoint, at machine speed. TestVibe treats that gap as the enemy. Nothing gets recorded that a cold replay can't reproduce: navigation and login live explicitly in the Background, every asserted outcome is proven against the live page at recording time, and each scenario starts from a clean context so scenario 2 never silently depends on scenario 1's leftovers.

The user-facing payoff is simple: "generation succeeded" means "this test passes," not "this test compiled." When you hit Run afterward, you're re-running something that has already survived the exact conditions of a real run.

One more artifact worth reading after success: the What the Assistant learned card. It lists app-specific behavior the agent discovered while exploring — a control that needs a beat before it's clickable, navigation it had to work around. It answers why the test is built the way it is, not just what it does.

When your app changes: regenerate, don't rewrite

Six weeks later the "Save" button becomes "Save changes" and the test goes red. This is where the spec-first design pays rent: the Gherkin is the durable asset, and the code is regenerable.

The workflow is deliberately boring:

1. Open the feature and review the failed run's evidence (screenshots, trace, failed step).
2. If the behavior changed, edit the Gherkin to match reality. If only the UI surface changed, the spec may already be fine.
3. Hit Regenerate. The agent re-explores the live app — the current app, with the renamed button — and produces fresh, re-verified code against it.

The docs are blunt about one anti-pattern: don't regenerate blindly in a loop. If the output missed your intent, the fix is almost always an input fix — a clearer label in a step, a stronger Then, a missing secret, a Background for setup — then one regeneration. And since regeneration can replace generated files, hand-edits to generated code are better promoted into clearer Gherkin, where they'll survive the next regeneration too.

The pipeline in one sentence

Describe the behavior → review a readable Gherkin spec → an agent explores your real app in an isolated cloud session → it writes web-first Playwright code and verifies it against the live site, end to end, before you ever see "succeeded" → and when your app changes, you regenerate from the spec instead of hand-patching selectors.

The spec stays yours and stays readable. The code stays disposable and stays green. That's the trade TestVibe is built around — and it's why the tests actually pass. Want to see it on your own app? Get early access, or read more in the generation docs.