Skip to content
InnoQualis InnoQualis Docs

Runbook — BackendDown

Alert

Section titled “Alert”

BackendDownSyntheticProbe (Spec 30.1 — synthetic acceptance probe) and the eventual BackendDown (Spec 30.3 — production SLO rule) both link here. The two rules share a runbook because their response procedure is identical.

Severity

Section titled “Severity”

critical

Audience

Section titled “Audience”

ops (synthetic probe) / both (production SLO rule, when 30.3 lands)

Symptom

Section titled “Symptom”

up{job="innoqualis-backend"} == 0 sustained — Prometheus has not received a successful scrape from the backend service for ≥1 minute.

When this fires from the production SLO rule (post-30.3), the customer-visible symptom is full backend outage: /api/* endpoints fail; the Next.js frontend stays stuck on loading spinners or shows “Network Error” toasts; integrations 5xx.

When this fires from the Spec 30.1 synthetic probe, it could be either a real outage OR a Prometheus-side issue (scrape config / network). Confirm which before reaching for the production playbook.

Impact

Section titled “Impact”
  • Production SLO rule firing: full outage for all tenants. Background workers stalled. AI features down. This is the audience-both case — tenant admins are also notified per ADR 0033.
  • Synthetic probe firing without a real outage: no customer impact; the alert is acting as the canary for the routing pipeline itself (Spec 30.1 acceptance criterion: “synthetic test alert arrives at contact@innoqualis.com within 60 seconds”).

Diagnosis

Section titled “Diagnosis”

  1. Is the backend actually down, or is this a Prometheus-side issue?

    Terminal window
    curl -fsS https://hub.innoqualis.com/api/status | jq

    If this returns 200 with a healthy payload, the backend is up — the alert is firing because Prometheus can’t reach it (network / scrape config). Move to step 3.

  2. If the curl fails, run the InnoQualisBackendDown diagnosis — they share the same root-cause tree (container down / crashing / unreachable).

  3. If the curl succeeds, the scrape pipeline is broken. Check:

    Terminal window
    docker logs --tail 100 innoqualis-prometheus | grep -iE 'scrape|target|error'
    curl -fsS http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.labels.job=="innoqualis-backend")'

    Either the backend’s /metrics endpoint is returning errors, or Prometheus can’t reach the backend container on the docker network.

Mitigation

Section titled “Mitigation”
  • Real outage → follow InnoQualisBackendDown mitigation.
  • Scrape-pipeline issue → restart Prometheus (docker compose -f monitoring/docker-compose.monitoring.yml restart prometheus); verify the docker network includes both innoqualis-backend and innoqualis-prometheus.

Resolution

Section titled “Resolution”
  • Bad deploy → rollback (git revert HEAD && make prod-up).
  • Migration failure → PodCrashLoopBackOff diagnosis.
  • DB / Redis / Qdrant outage → chase the corresponding runbook.
  • Scrape config drift → fix monitoring/prometheus.yml and reload.

Prevention

Section titled “Prevention”
  • Two-tier alerting: BackendDownSyntheticProbe (1m for) for fast acceptance + BackendDown (2m for, when 30.3 lands) for production SLO. The synthetic probe doubles as the canary for the routing pipeline itself — proves the email path works.
  • Health-check tuning per InnoQualisBackendDown runbook.
  • Multi-instance backend (Phase 26 cloud migration) so a single container failure isn’t a full outage.
Section titled “Related”

Last reviewed

Section titled “Last reviewed”

2026-05-29 — ops (initial runbook, shipped with Spec 30.8 to cover 30.1’s synthetic probe link)