Operations
Backup and restore
On the appliance, the data disk is the backup unit and everything else is a download away. On Compose and Kubernetes there are two pieces — the database and the object store — and they must be captured together. Procedures for all three are below.
What has to be backed up
The same split in all three shapes: the data in one place, and the key that decrypts it in another. A backup missing the second one is a backup of ciphertext.
| Shape | Back up | Because |
|---|---|---|
| Appliance | The data disk (disk 2), in full. | It holds the database, all artifacts and traces, this install's generated secrets, its install identity and its license. The OS disk holds nothing of yours. |
| Compose | Both named volumes: testvibe_pgdata and
testvibe_blobdata. Keep your .env too. |
The database volume holds your data and the key that decrypts it.
The blob volume holds the artifacts the database points at. Back them up
together. .env holds the keys that sign object-store links —
without it, existing links break. |
| Kubernetes | The database (yours or the chart's PVC), the object-store PVC, and the Secret. | Same split. Again the database carries its own protection key; the Secret carries the blob signing and admin keys and the console password. |
TESTVIBE_SECRET_PROTECTION_KEY cannot be regenerated,
reconstructed or recovered by anyone — including us. It encrypts every
stored project secret and the durable sign-in cookies. Lose it and those are gone.
.env or your Secret. The app mints its own key into its own
database the first time it needs one, and both the app and the console read
it back from there. .env.example ships the variable
commented out, and the chart's values.yaml says
“LEAVE THIS EMPTY… your database backup already contains it”.
So: back up the database and you have backed up the key. There is nothing separate to keep, and nothing for the two containers to disagree about.
The exception is an install that was set up with an explicit key — if
TESTVIBE_SECRET_PROTECTION_KEY has a real value in your
.env or Secret, that value still wins and you must still keep it. Check
before you assume. And on Kubernetes, if you set one, pass it on
every helm upgrade: Helm does not remember it.
/etc/testvibe/app-secrets.env on the data disk. Back up the data disk
and you have it. Keeping a second copy of that one line off the machine is cheap
insurance.
Taking the backup
One procedure per shape. Whichever you use, the database and the object store must be captured at the same moment. They point at each other.
Appliance Appliance
- Shut the appliance down cleanly. A powered-off copy is a consistent copy, and the appliance is not a 24/7 transactional system — a nightly or weekly window is usually acceptable. If it is not, use your hypervisor's own application-consistent snapshot mechanism.
- Copy or snapshot the data disk — the whole
-data.vhdxor the OVA's second disk — to wherever your backups live. That one disk holds the database, every artifact, the secrets, the install identity and the license. - Separately, keep a copy of the secret-protection key off the
machine. It is one line of
/etc/testvibe/app-secrets.env; treat it like a root credential. Losing the disk and that copy is unrecoverable. - Record which release the appliance is on. A data disk restores onto an OS disk of the same version or newer — never older, because migrations are one-way.
Docker Compose Compose
Run these from the directory holding your docker-compose.yml.
- Dump the database. This is safe while the stack is running —
pg_dumptakes a consistent snapshot of its own.Success looks like: the command exits silently and theshelldocker compose exec -T postgres \ pg_dump -U tvadmin -Fc testvibe > testvibe-db-$(date +%F).dump
.dumpfile is megabytes, not zero bytes. - Copy the object store. A read-only helper container tars the
blob volume out to the current directory.
Success looks like: ashell
docker run --rm \ -v testvibe_blobdata:/data:ro \ -v "$PWD":/backup \ alpine tar -czf /backup/testvibe-blob-$(date +%F).tar.gz -C /data .
.tar.gzwhose size is in the same ballpark asdocker system df -v | grep testvibe_blobdatareports. - Do those two back to back, with no runs in flight if you can manage it. A gap between them is the window in which a run finishes, writes its trace, and lands in one backup but not the other.
- Keep your
.envwith them. It holdsTV_BLOBSERVER_SIGNING_KEY,TV_BLOBSERVER_ADMIN_KEY,TV_RUNNER_KEY,TV_PG_PASSWORDandTV_ADMIN_PASSWORD. The secret-protection key is not there — it is inside the database dump you just took. - Record the release you are on — the image tags in your
docker-compose.yml. You restore onto that version or newer, never older.
testvibe_pgdata taken from a live database is not
a consistent backup and may not restore. If you want the volume rather than a dump,
docker compose stop first, tar it the same way as the blob volume, then
docker compose start.
Kubernetes Kubernetes
Written for a release called testvibe in namespace
testvibe. If you brought your own database
(postgresql.enabled=false), back that up with your provider's tooling
and only do step 2 here.
- Dump the database. Safe while running.
Success looks like: ashell
kubectl -n testvibe exec statefulset/testvibe-postgresql -- \ pg_dump -U tvadmin -Fc testvibe > testvibe-db-$(date +%F).dump
.dumpfile that is not zero bytes. - Capture the object-store PVC
(
testvibe-blobdata). If your storage class supports snapshots, that is the clean way and needs no downtime:Success looks like:shellkubectl -n testvibe apply -f - <<'EOF' apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: testvibe-blobdata-snap spec: volumeSnapshotClassName: <your-snapshot-class> source: persistentVolumeClaimName: testvibe-blobdata EOFkubectl -n testvibe get volumesnapshotshowsREADYTOUSE true. - No snapshot class? Copy it out instead. The PVC is
ReadWriteOnce, so the blob server has to let go of it first. This is downtime for artifacts and traces — the app itself keeps running.Success looks like: the blob server is back toshellkubectl -n testvibe scale deploy/testvibe-blobserver --replicas=0 # mount testvibe-blobdata into a throwaway pod and tar /data out kubectl -n testvibe scale deploy/testvibe-blobserver --replicas=1
1/1and a past run's trace still opens. - Back up the Secret
(
testvibe-secrets, or your own if you setsecrets.existingSecret), and keep yourvalues.yamlwith it.That file contains live credentials in base64 — store it where you store passwords, not next to the dump.shellkubectl -n testvibe get secret testvibe-secrets -o yaml > testvibe-secrets.yaml
- Record the chart version and image tag you are on. Restore onto that or newer, never older.
How to restore
The rule that governs all three: restore the database and the object store from the same backup run, and bring the database up first.
Neither failure announces itself at restore time. You find out days later, when somebody opens a trace.
Appliance Appliance
- Build a fresh appliance VM from the release you recorded (or a newer one), following the install steps — but attach your restored data disk instead of the release's empty one.
- Power on. The appliance sees an install that is already sealed, keeps its identity, applies any migrations the newer OS disk brings, and comes up as the same install — same install ID, same license, same data.
- Verify by signing in to the console and checking the License panel still shows your install ID as licensed, then open a past run and confirm its trace loads.
The pairing problem cannot happen here: the database and the object store are both on the one disk you restored, so they are always from the same instant.
Docker Compose Compose
- Start from a stopped stack and empty volumes.
That deletes data. Only do it when you mean to replace what is there.shell
docker compose down docker volume rm testvibe_pgdata testvibe_blobdata
- Restore the object store first. It is inert — nothing reads it
until the app is up.
shell
docker volume create testvibe_blobdata docker run --rm \ -v testvibe_blobdata:/data \ -v "$PWD":/backup \ alpine tar -xzf /backup/testvibe-blob-<date>.tar.gz -C /data
- Bring up the database and the schema, and nothing else.
Success looks like:shell
docker compose up -d postgres docker compose exec postgres pg_isready -U tvadmin -d testvibe
accepting connections. - Load the dump.
Success looks like: it finishes with noshell
docker compose exec -T postgres \ pg_restore -U tvadmin -d testvibe --clean --if-exists < testvibe-db-<date>.dump
errorlines. Harmlessdoes not exist, skippingnotices from--cleanon a fresh database are expected. - Now start everything else. The migrate job runs and brings the
restored schema up to the version of the images you are on.
Success looks like:shell
docker compose up -d docker compose ps
webandblobserverreporthealthy, andmigrateshowsExited (0). - Put your original
.envback before that last step if you have not already. A differentTV_BLOBSERVER_SIGNING_KEYdoes not lose data, but it invalidates every link already handed out. - Verify. Sign in, open a run from before the backup, and confirm its Playwright trace loads. That single check proves the database and the object store agree.
Kubernetes Kubernetes
- Restore the object store first. From a VolumeSnapshot, create
the PVC from it (
dataSourcepointing at the snapshot) and name ittestvibe-blobdata. From a tar, scaledeploy/testvibe-blobserverto 0, unpack into the PVC, scale back to 1. - Install or scale up the chart with the app stopped so the
database exists but nothing is writing to it:
Pass the sameshell
helm upgrade --install testvibe oci://testvibe.azurecr.io/charts/testvibe \ --set web.replicaCount=0 --set admin.enabled=false kubectl -n testvibe rollout status statefulset/testvibe-postgresql
secrets.*values you always pass. - Load the dump.
shell
kubectl -n testvibe exec -i statefulset/testvibe-postgresql -- \ pg_restore -U tvadmin -d testvibe --clean --if-exists < testvibe-db-<date>.dump
- Bring the app back. The migrations Job and each app's
wait-for-schemainit container handle the ordering for you.shellhelm upgrade testvibe oci://testvibe.azurecr.io/charts/testvibe \ --set web.replicaCount=1 --set admin.enabled=true kubectl -n testvibe rollout status deploy/testvibe-web
- Restore the Secret if this is a new cluster
(
kubectl -n testvibe apply -f testvibe-secrets.yaml), so the blob signing key and console password match what the data expects. - Verify. Open a run from before the backup and confirm its trace loads.