Operations

Backup and restore

On the appliance, the data disk is the backup unit and everything else is a download away. On Compose and Kubernetes there are two pieces — the database and the object store — and they must be captured together. Procedures for all three are below.

01

What has to be backed up

The same split in all three shapes: the data in one place, and the key that decrypts it in another. A backup missing the second one is a backup of ciphertext.

ShapeBack upBecause
Appliance The data disk (disk 2), in full. It holds the database, all artifacts and traces, this install's generated secrets, its install identity and its license. The OS disk holds nothing of yours.
Compose Both named volumes: testvibe_pgdata and testvibe_blobdata. Keep your .env too. The database volume holds your data and the key that decrypts it. The blob volume holds the artifacts the database points at. Back them up together. .env holds the keys that sign object-store links — without it, existing links break.
Kubernetes The database (yours or the chart's PVC), the object-store PVC, and the Secret. Same split. Again the database carries its own protection key; the Secret carries the blob signing and admin keys and the console password.
The one unrecoverable thing TESTVIBE_SECRET_PROTECTION_KEY cannot be regenerated, reconstructed or recovered by anyone — including us. It encrypts every stored project secret and the durable sign-in cookies. Lose it and those are gone.
On Compose and Kubernetes it is already in your database backup You no longer set that key, and it is not in your .env or your Secret. The app mints its own key into its own database the first time it needs one, and both the app and the console read it back from there. .env.example ships the variable commented out, and the chart's values.yaml says “LEAVE THIS EMPTY… your database backup already contains it”.

So: back up the database and you have backed up the key. There is nothing separate to keep, and nothing for the two containers to disagree about.

The exception is an install that was set up with an explicit key — if TESTVIBE_SECRET_PROTECTION_KEY has a real value in your .env or Secret, that value still wins and you must still keep it. Check before you assume. And on Kubernetes, if you set one, pass it on every helm upgrade: Helm does not remember it.
Appliance The appliance is different, and still generates the key on first boot into /etc/testvibe/app-secrets.env on the data disk. Back up the data disk and you have it. Keeping a second copy of that one line off the machine is cheap insurance.
02

Taking the backup

One procedure per shape. Whichever you use, the database and the object store must be captured at the same moment. They point at each other.

Appliance Appliance

  1. Shut the appliance down cleanly. A powered-off copy is a consistent copy, and the appliance is not a 24/7 transactional system — a nightly or weekly window is usually acceptable. If it is not, use your hypervisor's own application-consistent snapshot mechanism.
  2. Copy or snapshot the data disk — the whole -data.vhdx or the OVA's second disk — to wherever your backups live. That one disk holds the database, every artifact, the secrets, the install identity and the license.
  3. Separately, keep a copy of the secret-protection key off the machine. It is one line of /etc/testvibe/app-secrets.env; treat it like a root credential. Losing the disk and that copy is unrecoverable.
  4. Record which release the appliance is on. A data disk restores onto an OS disk of the same version or newer — never older, because migrations are one-way.

Docker Compose Compose

Run these from the directory holding your docker-compose.yml.

  1. Dump the database. This is safe while the stack is running — pg_dump takes a consistent snapshot of its own.
    shell
    docker compose exec -T postgres \
      pg_dump -U tvadmin -Fc testvibe > testvibe-db-$(date +%F).dump
    Success looks like: the command exits silently and the .dump file is megabytes, not zero bytes.
  2. Copy the object store. A read-only helper container tars the blob volume out to the current directory.
    shell
    docker run --rm \
      -v testvibe_blobdata:/data:ro \
      -v "$PWD":/backup \
      alpine tar -czf /backup/testvibe-blob-$(date +%F).tar.gz -C /data .
    Success looks like: a .tar.gz whose size is in the same ballpark as docker system df -v | grep testvibe_blobdata reports.
  3. Do those two back to back, with no runs in flight if you can manage it. A gap between them is the window in which a run finishes, writes its trace, and lands in one backup but not the other.
  4. Keep your .env with them. It holds TV_BLOBSERVER_SIGNING_KEY, TV_BLOBSERVER_ADMIN_KEY, TV_RUNNER_KEY, TV_PG_PASSWORD and TV_ADMIN_PASSWORD. The secret-protection key is not there — it is inside the database dump you just took.
  5. Record the release you are on — the image tags in your docker-compose.yml. You restore onto that version or newer, never older.
Do not tar the pgdata volume while it is running A file-level copy of testvibe_pgdata taken from a live database is not a consistent backup and may not restore. If you want the volume rather than a dump, docker compose stop first, tar it the same way as the blob volume, then docker compose start.

Kubernetes Kubernetes

Written for a release called testvibe in namespace testvibe. If you brought your own database (postgresql.enabled=false), back that up with your provider's tooling and only do step 2 here.

  1. Dump the database. Safe while running.
    shell
    kubectl -n testvibe exec statefulset/testvibe-postgresql -- \
      pg_dump -U tvadmin -Fc testvibe > testvibe-db-$(date +%F).dump
    Success looks like: a .dump file that is not zero bytes.
  2. Capture the object-store PVC (testvibe-blobdata). If your storage class supports snapshots, that is the clean way and needs no downtime:
    shell
    kubectl -n testvibe apply -f - <<'EOF'
    apiVersion: snapshot.storage.k8s.io/v1
    kind: VolumeSnapshot
    metadata:
      name: testvibe-blobdata-snap
    spec:
      volumeSnapshotClassName: <your-snapshot-class>
      source:
        persistentVolumeClaimName: testvibe-blobdata
    EOF
    Success looks like: kubectl -n testvibe get volumesnapshot shows READYTOUSE true.
  3. No snapshot class? Copy it out instead. The PVC is ReadWriteOnce, so the blob server has to let go of it first. This is downtime for artifacts and traces — the app itself keeps running.
    shell
    kubectl -n testvibe scale deploy/testvibe-blobserver --replicas=0
    # mount testvibe-blobdata into a throwaway pod and tar /data out
    kubectl -n testvibe scale deploy/testvibe-blobserver --replicas=1
    Success looks like: the blob server is back to 1/1 and a past run's trace still opens.
  4. Back up the Secret (testvibe-secrets, or your own if you set secrets.existingSecret), and keep your values.yaml with it.
    shell
    kubectl -n testvibe get secret testvibe-secrets -o yaml > testvibe-secrets.yaml
    That file contains live credentials in base64 — store it where you store passwords, not next to the dump.
  5. Record the chart version and image tag you are on. Restore onto that or newer, never older.
03

How to restore

The rule that governs all three: restore the database and the object store from the same backup run, and bring the database up first.

Why the pairing matters The database holds the run rows. The object store holds the trace and screenshot files those rows point at. Restore a newer database next to an older object store and you get runs whose traces render as broken links — the app is asking for files that backup never contained. Restore it the other way round and you get files nothing references, quietly using disk forever.

Neither failure announces itself at restore time. You find out days later, when somebody opens a trace.

Appliance Appliance

  1. Build a fresh appliance VM from the release you recorded (or a newer one), following the install steps — but attach your restored data disk instead of the release's empty one.
  2. Power on. The appliance sees an install that is already sealed, keeps its identity, applies any migrations the newer OS disk brings, and comes up as the same install — same install ID, same license, same data.
  3. Verify by signing in to the console and checking the License panel still shows your install ID as licensed, then open a past run and confirm its trace loads.

The pairing problem cannot happen here: the database and the object store are both on the one disk you restored, so they are always from the same instant.

Docker Compose Compose

  1. Start from a stopped stack and empty volumes.
    shell
    docker compose down
    docker volume rm testvibe_pgdata testvibe_blobdata
    That deletes data. Only do it when you mean to replace what is there.
  2. Restore the object store first. It is inert — nothing reads it until the app is up.
    shell
    docker volume create testvibe_blobdata
    docker run --rm \
      -v testvibe_blobdata:/data \
      -v "$PWD":/backup \
      alpine tar -xzf /backup/testvibe-blob-<date>.tar.gz -C /data
  3. Bring up the database and the schema, and nothing else.
    shell
    docker compose up -d postgres
    docker compose exec postgres pg_isready -U tvadmin -d testvibe
    Success looks like: accepting connections.
  4. Load the dump.
    shell
    docker compose exec -T postgres \
      pg_restore -U tvadmin -d testvibe --clean --if-exists < testvibe-db-<date>.dump
    Success looks like: it finishes with no error lines. Harmless does not exist, skipping notices from --clean on a fresh database are expected.
  5. Now start everything else. The migrate job runs and brings the restored schema up to the version of the images you are on.
    shell
    docker compose up -d
    docker compose ps
    Success looks like: web and blobserver report healthy, and migrate shows Exited (0).
  6. Put your original .env back before that last step if you have not already. A different TV_BLOBSERVER_SIGNING_KEY does not lose data, but it invalidates every link already handed out.
  7. Verify. Sign in, open a run from before the backup, and confirm its Playwright trace loads. That single check proves the database and the object store agree.

Kubernetes Kubernetes

  1. Restore the object store first. From a VolumeSnapshot, create the PVC from it (dataSource pointing at the snapshot) and name it testvibe-blobdata. From a tar, scale deploy/testvibe-blobserver to 0, unpack into the PVC, scale back to 1.
  2. Install or scale up the chart with the app stopped so the database exists but nothing is writing to it:
    shell
    helm upgrade --install testvibe oci://testvibe.azurecr.io/charts/testvibe \
      --set web.replicaCount=0 --set admin.enabled=false
    kubectl -n testvibe rollout status statefulset/testvibe-postgresql
    Pass the same secrets.* values you always pass.
  3. Load the dump.
    shell
    kubectl -n testvibe exec -i statefulset/testvibe-postgresql -- \
      pg_restore -U tvadmin -d testvibe --clean --if-exists < testvibe-db-<date>.dump
  4. Bring the app back. The migrations Job and each app's wait-for-schema init container handle the ordering for you.
    shell
    helm upgrade testvibe oci://testvibe.azurecr.io/charts/testvibe \
      --set web.replicaCount=1 --set admin.enabled=true
    kubectl -n testvibe rollout status deploy/testvibe-web
  5. Restore the Secret if this is a new cluster (kubectl -n testvibe apply -f testvibe-secrets.yaml), so the blob signing key and console password match what the data expects.
  6. Verify. Open a run from before the backup and confirm its trace loads.
Test the restore before you need it. On the appliance, a restored data disk on a fresh OS disk is exactly the same operation as an upgrade, so testing one tests the other. On Compose and Kubernetes, do the whole thing once into a scratch namespace or a second host — the pairing mistake above is invisible until you look for it.