Skip to content
InnoQualis InnoQualis Docs

Runbook — HighErrorRate

Alert

Section titled “Alert”

HighErrorRate

Severity

Section titled “Severity”

critical

Audience

Section titled “Audience”

both

Symptom

Section titled “Symptom”

rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05 — more than 5% of HTTP responses are 5xx. Customers will be seeing failures on at least one workflow; the control panel /alerts page is red.

Impact

Section titled “Impact”
  • All tenants potentially affected (the rate is global, not per-tenant).
  • Customers see “Something went wrong” toasts in the UI, request retries, frustration.
  • This is audience: both — tenant admins are emailed too, not just ops.

Diagnosis

Section titled “Diagnosis”

  1. Which endpoint is failing?

    Terminal window
    docker logs --tail 1000 innoqualis-backend 2>&1 | grep -E '"status": 5' | jq -r '.path' | sort | uniq -c | sort -rn | head

    Pinpoints the URL pattern. If a single endpoint accounts for >80%, that’s the broken one.

  2. What’s the error?

    Terminal window
    docker logs --tail 1000 innoqualis-backend 2>&1 | grep -E 'ERROR|Traceback' | tail -30

    Read the most recent stack trace. Common patterns: OperationalError (DB), ConnectionError (Redis / Qdrant), KeyError (deploy issue with stale code expecting a field that doesn’t exist).

  3. What changed?

    Terminal window
    git -C /opt/innoqualis log --oneline --since='2 hours ago'

    A feat: or refactor: commit landed in the last 2 hours is the most common cause.

Mitigation

Section titled “Mitigation”
  • Recent deploy = roll back: git -C /opt/innoqualis revert HEAD --no-edit && make prod-up
  • DB / Redis / Qdrant outage: chase the corresponding runbook (DatabaseConnectionIssues, RedisConnectionIssues).
  • External service (Stripe, n8n, Listmonk): degrade gracefully — disable the relevant feature flag if available; if not, accept the partial outage and communicate via status page.

Resolution

Section titled “Resolution”
  • Code bug → hotfix PR with regression test in backend/tests/regressions/.
  • Capacity issue → restart + scale; follow up with profiling.
  • Downstream outage → restore the dependency; verify error rate returns to baseline.

Prevention

Section titled “Prevention”
  • Per-endpoint 5xx alert (this rule is global; a per-endpoint version would catch broken endpoints earlier).
  • Canary deploys (Phase 26 cloud migration).
  • Sentry exception tracking (Phase 28.2) — turns a “5xx fired” into “here’s the exact stack trace and tenant ID” without grepping logs.

Last reviewed

Section titled “Last reviewed”

2026-05-29 — ops