Skip to content
InnoQualis InnoQualis Docs

Runbook — InnoQualisHighCPUUsage

Alert

Section titled “Alert”

InnoQualisHighCPUUsage

Severity

Section titled “Severity”

warning

Audience

Section titled “Audience”

ops

Symptom

Section titled “Symptom”

rate(process_cpu_seconds_total{job="innoqualis-backend"}[5m]) > 0.8 — backend is consistently using >80% of one CPU core. Note this is per-process CPU, not node-level (see HighCPUUsage for the node-level analogue).

Impact

Section titled “Impact”
  • Backend latency degraded.
  • AI / embedding endpoints noticeably slower.
  • May progress to HighResponseTime / HighErrorRate if the load persists.

Diagnosis

Section titled “Diagnosis”

Run the HighCPUUsage runbook diagnosis. Specifically for this alert:

  1. Identify hot endpoint:

    Terminal window
    docker logs --tail 2000 innoqualis-backend 2>&1 | grep -E '"path"' | jq -r .path | sort | uniq -c | sort -rn | head

  2. DB-side activity:

    Terminal window
    docker exec innoqualis-db psql -U postgres -d eqms -c "SELECT query, count(*) FROM pg_stat_activity GROUP BY query ORDER BY count DESC LIMIT 5;"

Mitigation

Section titled “Mitigation”
  • Restart backend if a worker is stuck: docker compose -f docker/compose.yml restart backend.
  • Pause a tenant’s connector if one tenant is dominating ingestion.

Resolution

Section titled “Resolution”

See the HighCPUUsage Resolution section.

Prevention

Section titled “Prevention”
  • Per-tenant LLM token budget (Spec 24.8) — already in place; tune limits if it’s a recurring trigger.
  • Sentry performance traces (Phase 28.2) — pinpoints the slow endpoint without log-grepping.

Last reviewed

Section titled “Last reviewed”

2026-05-29 — ops