Runbook — DeploymentFailed

Alert

DeploymentFailed

Severity

critical

Audience

ops

Symptom

deployment_status{status="failed"} == 1. The alert label deployment_id identifies the run. The CI / deploy log carries the failure reason.

Impact

• Best case: the rollback in the deploy script succeeded; running version is the previous one; impact = the new feature isn’t live yet.
• Worst case: the deploy half-applied — new code on the backend, old code on the frontend, or mid-migration. Customers see broken endpoints.

Diagnosis

1. Which version is actually running?

   curl -fsS https://hub.innoqualis.com/api/status | jq '.version'
   docker inspect innoqualis-backend --format '{{.Config.Image}}'

   Compare with the intended new version in the deploy log.

2. What failed?

   gh run view <run-id> --log-failed

   Or tail /var/log/innoqualis/deploy.log. Typical: migration timeout, image pull failure, health-check timeout.

3. Migration state?

   docker exec innoqualis-db psql -U postgres -d eqms -c "SELECT version_num FROM alembic_version;"

   Compare with head in the new code. Mismatch = mid-migration state — DANGEROUS.

Mitigation

• Half-applied deploy → full rollback:

  ssh ops@prod 'cd /opt/innoqualis && git reset --hard HEAD~1 && make prod-up'

• Mid-migration → either complete it (docker compose run --rm backend python -m alembic upgrade head) or roll back (alembic downgrade -1) — pick based on whether the new code or old code is currently serving.
• Communicate via status page if customer impact is observed.

Resolution

• Reproduce the deploy failure in staging.
• Fix the root cause, write a regression test for the deploy script (Phase 28 test methodology).
• Re-attempt deploy.

Prevention

• Blue/green or canary deploys (Phase 26).
• Migration dry-run in CI before any deploy.
• Atomic deploy script: deploy.sh must either fully succeed or fully roll back. No half-states.

Last reviewed

2026-05-29 — ops