Should you trust a test an AI wrote?

Not on faith. A test you didn't write, checking behavior you didn't verify, is a liability whether a human or a model produced it. The question isn't whether AI can be trusted to write tests. It's what has to be true before any test earns your trust, and the answer is the same for both: it has to have run, and someone has to have read what it claims.

That standard is actually good news for generated tests, because a model can run its own output in a loop far faster than you can. The trust doesn't come from the model being clever. It comes from execution.

Why unrun code is the real risk

A test is a claim: "when the user does X, the app does Y." Generated code makes that claim in fluent, confident Playwright. Fluency is exactly the problem. The output looks right, so it's tempting to skim it, see reasonable-looking selectors and assertions, and commit.

Here are the failure modes that survive a skim but die on first run:

• Selectors for elements that don't exist. The model inferred a data-testid or a button label from context that isn't in your DOM. The test never reaches its assertion.
• Assertions that can't fail. The classic is a test that clicks through a flow and then checks something trivially true, so it passes no matter what the app did.
• Tests that navigate but never verify. Lots of page.click(), page.fill(), and page.goto(), and no expect() that pins the actual outcome. Green, and meaningless.
• Waits tuned to a lucky run. A hard-coded waitForTimeout that happened to be long enough once, and will flake in CI under load.

A tautological assertion is the most dangerous because it stays green forever:

// Looks like a login test. Proves nothing.
test('user can log in', async ({ page }) => {
  await page.goto('/login');
  await page.getByLabel('Email').fill('user@example.com');
  await page.getByLabel('Password').fill('correct-horse');
  await page.getByRole('button', { name: 'Sign in' }).click();
  expect(page.url()).toBeTruthy(); // a URL always exists
});

That test passes against a completely broken login. The fix isn't more AI, it's an assertion tied to an observable outcome:

  await page.getByRole('button', { name: 'Sign in' }).click();
  await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible();
  await expect(page).toHaveURL(/\/dashboard/);

None of these bugs are exotic. They're the normal failure surface of code generated from an intent rather than observed from a running system. Which is exactly why running the code is the thing that separates a real test from a plausible one.

Verification is what converts a draft into a test

The move that makes a generated test trustworthy is boring and mechanical: run it against the real application and see what happens. A selector that doesn't exist throws. An assertion that can't fail also can't tell you the app is broken, so you check it against a known-good and known-bad state. A flow that never asserts produces a green run that a reviewer can spot as suspiciously fast and empty.

This is where a generation system that closes the loop earns its keep. In TestVibe, generation doesn't just emit code. The agent explores the target site, writes the Playwright for each scenario, and then replays what it wrote against the live site. A failed replay loops back to writing and tries again, so a scenario that references a button that isn't there gets caught and rewritten before you ever see it. After every scenario is recorded, the whole assembled spec is run end-to-end one more time. As the docs put it, "Recording every step is not the same as the finished test passing a real run — this is the check that proves it does." A feature only reaches generated status once that final verification passes, and there's no way to skip it.

That's the honest version of "AI writes your tests." Not the model is trustworthy, but the output was executed against your actual app before it was handed to you. The green checkmark means a browser really did the thing.

The human checkpoint is the spec, not the code

Execution catches code that doesn't run. It cannot catch code that runs and verifies the wrong thing. A test can pass perfectly while asserting a behavior you never wanted. That gap is where a human belongs, and the cheapest place to stand is the spec.

Read the Gherkin first. If the feature description is wrong, the generated code will faithfully, verifiably test the wrong behavior. TestVibe's own guidance is blunt about this: AI generation does not replace review, and you still decide whether the spec describes the right behavior, whether the generated code checks the right outcome, and whether a failure means the app is broken or the test is wrong.

A quick spec-and-code pass, before you lean on the test in CI:

• Does the scenario describe behavior you actually care about? A verified test of an irrelevant path is verified waste.
• Is there a real assertion, and does it pin the outcome? Look for an expect on visible state, not on the fact that a page loaded.
• Does the code follow the journey the spec describes? Same story, told twice — Gherkin and code should agree.
• Is the test data repeatable and free of accidental private data? Credentials belong in configured secrets, not baked into the spec.

You don't need to understand every line to catch the obvious wrong turns. You're not re-deriving the test. You're confirming the intent is right, and letting execution confirm the mechanics.

Trust, allocated correctly

So: should you trust a test an AI wrote? Trust the ones that ran green against your real app and whose spec you read and agreed with. Distrust the ones that were merged on the strength of looking reasonable — and hold human-written tests to precisely the same bar, because an unrun assertion is worthless no matter who typed it.

The shift AI actually enables isn't blind trust. It's moving your attention up a level: from writing selectors and waits by hand to reviewing intent, while a machine handles the tedious, essential work of running the thing until it genuinely passes. See how verification works in the docs, or get early access to try generation that has to run before it ships.