Skip to content
InnoQualis InnoQualis Docs

Runbook — BackupJobFailed

Alert

Section titled “Alert”

BackupJobFailed

Severity

Section titled “Severity”

critical

Audience

Section titled “Audience”

ops

Symptom

Section titled “Symptom”

The Prometheus metric backup_job_status{status="failed"} == 1 — meaning whichever backup script most recently reported in did so as failed. The alert label job_id identifies the run.

Impact

Section titled “Impact”
  • No immediate customer impact — the system is still serving.
  • Recovery posture is degraded: if the next disaster strikes before the next successful backup, recovery point objective slips by the missed window.
  • Compliance posture (21 CFR Part 11): documented procedure must be visibly executed; failed backups need investigation logged.

Diagnosis

Section titled “Diagnosis”

  1. Which backup ran and what was the failure?

    Terminal window
    ssh ops@<host> 'tail -100 /var/log/innoqualis/backup.log'

    Typical failure modes: source DB unreachable, destination volume full, S3 credentials expired (post-cloud), pg_dump version mismatch.

  2. Was anything successful recently?

    Terminal window
    ssh ops@<host> 'ls -lt /backups/ | head -10'

    Confirm the last successful backup timestamp.

  3. Is the source / destination healthy?

    • Source DB: docker exec innoqualis-db psql -U postgres -c 'SELECT 1;'
    • Destination: df -h /backups

Mitigation

Section titled “Mitigation”
  • Retry the backup manually once the failure mode is understood:
    Terminal window
    ssh ops@<host> 'cd /opt/innoqualis && bash scripts/backup.sh'
  • If the destination is full: free space per the DiskSpaceLow runbook, then retry.

Resolution

Section titled “Resolution”
  • Persistent failure → re-evaluate the backup pipeline. Most common root cause pre-cloud: backup script silently broken by a refactor without a regression test.
  • Add a retention policy if the destination keeps filling.
  • Post-cloud (Phase 26): backups go to S3 with lifecycle rules.

Prevention

Section titled “Prevention”
  • Backup verification test (Phase 29 go-live checklist) — periodically restore the latest backup to a sandbox and assert it boots. Detects silent corruption / bad scripts.
  • Backup freshness alert — alert if no successful backup in 24h (this rule only catches failed — not missing).

Last reviewed

Section titled “Last reviewed”

2026-05-29 — ops