Runbook — InnoQualisBackendDown

Alert

InnoQualisBackendDown

Severity

critical

Audience

both

Symptom

up{job="innoqualis-backend"} == 0 for 2 minutes. Prometheus can’t scrape the backend’s /metrics endpoint. The control panel /alerts page shows the alert. Customers see the frontend stuck on loading spinners, or “Network Error” toasts.

Impact

• Full outage for all tenants. The backend is the only thing answering /api/*.
• This is audience: both — tenant admins also receive this email (per ADR 0033 audience model).
• Background workers also down (they run in the backend container).

Diagnosis

1. Is the container running?

   docker ps --filter name=innoqualis-backend --format 'table {{.Names}}\t{{.Status}}'

2. If running, can we hit it locally?

   docker exec innoqualis-backend curl -fsS http://localhost:8000/api/status

   If this succeeds → it’s a network / nginx problem, not a backend problem. Check docker logs innoqualis-nginx.

3. If crashing or restarting, see PodCrashLoopBackOff diagnosis — same steps from here.

Mitigation

• Container down → start it: docker compose -f docker/compose.yml up -d backend
• Container crashing → chase the crash cause via PodCrashLoopBackOff.
• Container healthy but unreachable from outside: reload nginx (docker compose restart nginx), verify the docker network (docker network inspect innoqualis-network should show both nginx and backend).

Resolution

• Bad deploy → rollback (git revert HEAD && make prod-up).
• DB / migration issue → see DatabaseConnectionIssues.
• Resource exhaustion → see HighCPUUsage / HighMemoryUsage.

Prevention

• Health-check tuning: the readiness probe should detect failure within 30s, not 2min.
• Auto-restart on health failure: docker restart: unless-stopped + health-check + --restart-on-unhealthy.
• Multi-instance backend (Phase 26 cloud migration) so a single container failure isn’t a full outage.

Last reviewed

2026-05-29 — ops