Runbook — BackupJobFailed

Alert

BackupJobFailed

Severity

critical

Audience

ops

Symptom

Either of two things paged you:

• Better Stack heartbeat missed — innoqualis-backup.sh pings BACKUP_HEARTBEAT_URL only as the very last statement of a fully successful run. No ping within the expected window (nightly + grace period) means the run either failed partway through or never fired at all (systemd timer didn’t run, box was down, etc.).
• The script itself exited non-zero — visible in journalctl -u innoqualis-backup.service or the log file below — even if you’re investigating this ahead of a heartbeat alert.

(KAN-792 shipped this as a heartbeat-only design, not a Prometheus metric. An earlier draft of this runbook described a backup_job_status metric that was never actually implemented — there is no such metric in monitoring/. Corrected 2026-09-12.)

Impact

• No immediate customer impact — the system is still serving.
• Recovery posture is degraded: if the next disaster strikes before the next successful backup, recovery point objective slips by the missed window (RPO is 24h on a good day — see spec/deployment.md’s Backups section).
• Compliance posture (21 CFR Part 11): documented procedure must be visibly executed; failed backups need investigation logged.

Diagnosis

1. Which step failed?

   ssh ops@<host> 'tail -100 /var/log/innoqualis/backup.log'
   # or, for the structured systemd view:
   ssh ops@<host> 'journalctl -u innoqualis-backup.service -n 200 --no-pager'

   Each line is ts=… step=… status=… msg=…; the failing step name and message are in the last status=error line. Typical failure modes: Postgres/Docker unreachable, a required env var missing from /root/innoqualis/.env, the Storage Box SSH key rotated/expired, local disk full ($BACKUP_LOCAL_DIR, default /var/backups/innoqualis), a Qdrant collection snapshot timing out.

2. Was anything successful recently?

   ssh ops@<host> 'ls -lt /var/backups/innoqualis | head -10'

   Confirm the last successful local staging directory’s date, and cross-check against what’s actually on the Storage Box (needs rclone configured the same way the backup script is):

   ssh ops@<host> 'rclone lsd hetzner:backups/daily/ --timeout 60s'

3. Is the source / destination healthy?

   • Source DB: docker exec innoqualis-db pg_isready -U postgres
   • Backend (needed for the Qdrant snapshot step): docker exec innoqualis-backend true
   • Local staging disk: df -h /var/backups/innoqualis
   • Wiring end to end, without doing a real backup:

     ssh ops@<host> '/usr/local/lib/innoqualis-backup/innoqualis-backup.sh --self-test'

Mitigation

• Retry the backup manually once the failure mode is understood:

  ssh ops@<host> '/usr/local/lib/innoqualis-backup/innoqualis-backup.sh'

  (the script’s own flock refuses to run if a prior attempt is still holding the lock — check ps aux | grep innoqualis-backup first if the retry itself refuses to start).
• If the destination is full: free space per the DiskSpaceLow runbook, then retry.
• If a credential rotated (Storage Box SSH key, B2 application key, age recipient): update /root/innoqualis/.env, re-render the rclone config (/usr/local/lib/innoqualis-backup/render-rclone-conf.sh), then retry.

Resolution

• Persistent failure → re-evaluate the backup pipeline. Most common root cause: a refactor of scripts/backup/innoqualis-backup.sh without running backend/tests/regressions/test_backup_scripts_static_kan792.py first, or a credential that expired without a rotation reminder.
• Retention is already policy-driven (BACKUP_LOCAL_KEEP/BACKUP_DAILY_KEEP/BACKUP_MONTHLY_KEEP — see scripts/backup/README.md); a full destination usually means one of those envs regressed to a much larger value, not that retention is missing.

Prevention

• Restore drill (scripts/backup/innoqualis-restore-drill.sh) — run at least monthly; it actually restores the latest backup into a throwaway Postgres container and checks row counts + alembic current. Detects silent corruption that a mere “the upload succeeded” check would miss. Record the report under validation/executed/.
• Heartbeat freshness — Better Stack alerts on a missed check-in on its own; there is nothing additional to configure here.

Last reviewed

2026-09-12 — ops (KAN-792 rewrite: matches the shipped Hetzner/B2/heartbeat implementation, not the earlier speculative Prometheus-metric draft)