Runbook — HighCPUUsage

Alert

HighCPUUsage

Severity

warning

Audience

ops

Symptom

Node-level CPU averaged across cores exceeds 80% idle-subtracted for more than 5 minutes. In the control panel /alerts page the alert shows the instance label of the affected node. Grafana’s “Node CPU” panel for that instance is solid red.

Customer-visible signal is usually mild — slower responses but rarely outright failures. Becomes critical if it correlates with HighResponseTime or HighErrorRate firing on the same instance.

Impact

Latency degradation across all tenants on the affected node.
AI embedding requests (Qdrant ingestion, RAG queries) noticeably slower.
Background workers (notification sender, document expiry scanner) may miss their schedule.
No data loss; nothing is hard-failing yet.

Diagnosis

Run all three in order — each tells you something the previous one didn’t.

Who is burning CPU on the host?
Terminal window
```
ssh ops@<instance> 'top -b -n 1 -o %CPU | head -20'
```
Read off the top 5 processes by %CPU. Look for python / uvicorn (backend), node (frontend / control), postgres, redis-server.
Is the backend the culprit?
Terminal window
```
docker stats --no-stream innoqualis-backend innoqualis-frontend innoqualis-db
```
Compares per-container CPU. If innoqualis-backend is >300% (3 of 4 vCPUs) the issue is in the FastAPI worker pool — almost always an AI / embedding loop or a pathological SELECT from an unindexed query.
What is the backend doing?
Terminal window
```
docker logs --tail 500 innoqualis-backend 2>&1 | grep -E 'duration_ms|qdrant|embedding|SELECT' | tail -50
```
Look for a single endpoint hit hundreds of times per minute, or duration_ms values trending upward, or a tenant ID that dominates the log lines.

If postgres is the top consumer, jump to the DatabaseConnectionIssues runbook — the root cause is usually a long-running query holding locks, not CPU per se.

Mitigation

Pick the one matching what diagnosis showed:

Backend worker storm: restart with the same worker count to drop in-flight requests.
Terminal window
```
docker compose -f docker/compose.yml restart backend
```
Single tenant flooding ingestion: pause that tenant’s connector via the control panel /admin/tenants/<id> page, or — if the control panel is also slow — UPDATE connectors SET state='paused' WHERE tenant_id=<id>.
Host genuinely under-provisioned (sustained >80% for hours, not a spike): vertical-scale the VPS (Hetzner control panel → resize). This is the only legitimate “buy more hardware” path; do not reach for it before you’ve ruled out the application causes above.

Resolution

Identify the root cause from diagnosis, then either:

Code path: open a ticket, link the offending endpoint, target it in the next patch release. Examples: an unbounded SELECT * FROM documents without tenant_id scoping, a for tenant in tenants loop calling the LLM serially.
Bad tenant input: contact the tenant via contact@innoqualis.com if the burn is from a misconfigured connector or oversized import. Document in tasks/lessons.md.
Capacity: file a Phase 26 (cloud migration) follow-up — pre-cloud, single-VPS capacity headroom is a known constraint.

Prevention

Per-tenant rate limit on AI endpoints — Phase 24.8 (KAN-289) already enforces a per-tenant LLM token budget; this should subsume the “one tenant burns CPU” failure mode once it’s tuned. If a tenant tripped the budget but CPU still burned, the budget cap needs lowering.
Background-worker queue depth alert — separate alert for queue depth >1000 would catch worker storms before CPU spikes. Not yet shipped; track in Phase 30 follow-ups.
Profile the hot endpoint — add an APM trace for any endpoint regularly seen in step 3 of diagnosis. Sentry performance traces in Phase 28.2 cover this once it lands.

Last reviewed

2026-05-29 — ops (initial runbook, shipped with Spec 30.8)