Telemetry: watch your app between test runs

A passing test suite tells you the paths you tested still work. It says nothing about the server underneath them — whether memory is creeping up run over run, whether an unhandled exception is quietly firing in production, whether the box is near its CPU ceiling. TestVibe telemetry closes that gap: install a small agent in the application you test, and its server-side vitals and errors show up in the same place as your test results.

What telemetry shows you

Once the agent is running, telemetry lands in two surfaces.

Dashboard → Live Servers is a real-time view. Each server reporting into your project renders as a card with live charts for CPU, memory, active sessions, memory per session, thread count, and managed GC heap. It streams only while you are watching and is not stored — open it while a load test runs and watch the server react as virtual users replay your features.

Load test results embed the same server vitals next to the client-side latency charts. Client numbers tell you what users experienced; server telemetry tells you why. A p95 that climbs while CPU saturates is a capacity ceiling, not a code bug.

The Telemetry Overview: CPU, memory, sessions, threads, and GC heap for a live server, with a server-info card and the browsers currently connected.

The overview also carries a Server info card — runtime, OS, CPU cores, environment, host names, process start time — and a Users panel that groups the browsers currently connected to the server by instance, with each tab's viewport, age, and open pages.

Wire up the agent

You need one thing to start: a workspace-scoped API key (tvb_…) from Settings → CLI & API keys. The agent discovers your projects from the key, so on .NET you don't even pass a project id.

For .NET, install the TestVibe.Telemetry package. It targets netstandard2.0, so it runs on .NET Framework 4.6.1+, .NET Core 2.0+, and .NET 5–10+.

dotnet add package TestVibe.Telemetry

Register it once at startup. The optional SessionCount callback is what unlocks the Sessions and Memory / session metrics — wire it to whatever represents an active session in your app (SignalR connections, signed-in users, a framework's built-in counter).

// Program.cs
using TestVibe;

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

Telemetry.Register(
    builder.Configuration["TestVibe:Server"] ?? "https://app.testvibe.com",
    builder.Configuration["TestVibe:ApiKey"], // tvb_… from Settings → CLI & API keys
    new TelemetryOptions
    {
        SessionCount = () => MyConnectionCounter.Current,
    });

app.Run();

Node services get a single-file agent instead. Copy testvibe-load-agent.js from Settings → Telemetry — it has no dependencies — and start it once at boot:

const { startTestVibeLoadAgent } = require('./testvibe-load-agent');

startTestVibeLoadAgent({
  server: 'https://app.testvibe.com',
  apiKey: 'tvb_…',                  // Settings → CLI & API keys
  project: '<your-project-id>',
  sessions: () => myActiveSessions, // optional
});

The agent is deliberately quiet. It only sends data while a load run is active or someone is watching the Dashboard — there is no idle traffic — and every send is best-effort, so it can never disturb the application it is measuring.

Custom metrics, info, and errors

Beyond the built-in vitals, the agent gives you three calls you can make from anywhere in your code.

// Gauges — charted alongside CPU/memory (latest value wins, up to 20 metrics).
TestVibe.Telemetry.ReportMetric("Orders/min", ordersPerMinute);

// Static facts shown in the server card header.
TestVibe.Telemetry.ReportInfo("Region", "westeurope");

// Errors — grouped and persisted under Dashboard → Telemetry → Errors & exceptions.
try { DoWork(); }
catch (Exception ex) { TestVibe.Telemetry.ReportError(ex); throw; }

Unhandled exceptions are captured automatically; ReportError lets you add the handled ones you'd otherwise never see. Unlike the live vitals, errors are grouped and persisted, so you can review what fired during a run long after the run finished.

Reading the signal

A few patterns are worth training your eye on:

CPU drifting toward saturation while latency p95 rises is the classic capacity ceiling. Scale or optimize before you loosen a threshold.
Memory that climbs run over run without recovering points at a leak the load is exposing. Memory / session isolates whether the growth is per-session or global.
Sessions confirms the load actually produced the concurrency you configured — a suspiciously low count usually means your virtual users aren't establishing real sessions.

From a telemetry signal to a covering test

Telemetry is most useful when it feeds back into your suite. Say the Errors & exceptions panel surfaces a NullReferenceException on checkout that only appears under concurrency — a path none of your current tests exercise. That's the moment to write a test for it.

In TestVibe you don't hand-write the Playwright. Create a feature and describe the flow in plain language, keyed off the visible labels involved:

Feature: Checkout with an empty saved address

  Background:
    Given I am on the "Sign in" page
    When I sign in as a customer with no saved address
    And I add "Sauce Labs Backpack" to the cart

  Scenario: Placing an order when no address is on file
    When I open the cart and click "Checkout"
    And I click "Continue" without selecting an address
    Then an "Add a shipping address" message should appear
    And the order should not be submitted

TestVibe generates the Playwright test from that description and runs it in a cloud sandbox to confirm it actually reproduces — a generated test that hasn't been executed is just a guess, so verification is part of generation. Now the crash you saw in telemetry has a test that fails until it's fixed and passes forever after. Telemetry found the gap; the test closes it.

Where telemetry lives

Persisted telemetry — the errors and exception history — counts against your workspace's shared storage quota, alongside runs, load results, and generated tests. You can set how long TestVibe keeps each data type in Settings → Storage, where a per-type retention policy evicts the oldest entries first as you approach the quota. Live Servers data isn't stored at all, so it costs nothing to keep an eye on.

Wiring up the agent takes one package and one line at startup, and it turns your test suite from a snapshot into a continuous read on the app behind it. See the Server telemetry docs for every framework snippet, or get early access to try it against your own app.