Skip to content
InnoQualis InnoQualis Docs

Runbook — HighMemoryUsage

Alert

Section titled “Alert”

HighMemoryUsage

Severity

Section titled “Severity”

warning

Audience

Section titled “Audience”

ops

Symptom

Section titled “Symptom”

Node-level memory utilisation (1 - MemAvailable / MemTotal) exceeds 85% for more than 5 minutes. Grafana’s “Node Memory” panel shows used+buffers+cached climbing without recovery. The instance label on the alert names the affected host.

If memory crosses ~95% and the kernel starts swapping, you’ll see PodCrashLoopBackOff fire next as the OOM killer takes containers down.

Impact

Section titled “Impact”
  • No customer-visible failures yet, but a cliff is near — once the OOM killer fires, the backend (and any tenant request in flight) dies.
  • AI / embedding queries get slower as the kernel pages out hot caches.
  • Future container restarts will be slower (cold caches).

This alert is a warning specifically so on-call has time to act before it becomes InnoQualisBackendDown.

Diagnosis

Section titled “Diagnosis”

  1. Who is holding the memory?

    Terminal window
    ssh ops@<instance> 'ps aux --sort=-%mem | head -15'

    Top entries are almost always: Python workers (uvicorn), PostgreSQL backends, Qdrant binary, Redis. A single Python worker over 2 GB is a leak.

  2. Per-container breakdown.

    Terminal window
    docker stats --no-stream --format 'table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}'

    Compare to the baseline: backend usually ~800 MB, frontend ~400 MB, db ~1.5 GB, qdrant ~600 MB. Anything 2× baseline is the suspect.

  3. Is it the backend?

    Terminal window
    docker exec innoqualis-backend ps -o pid,rss,cmd -C python | sort -k2 -n

    Tells you per-worker RSS. If one worker is 3× the others, that worker has the leak — restart isolates it.

If postgres is the top consumer, check SELECT pg_size_pretty(sum(temp_bytes)) FROM pg_stat_database; — runaway work_mem from an unconstrained query.

Mitigation

Section titled “Mitigation”
  • Restart the leaky service — the fastest path to free memory.
    Terminal window
    docker compose -f docker/compose.yml restart backend
    Backend restart is graceful (uvicorn drains in-flight requests up to 30s).
  • Drop the Qdrant page cache if Qdrant is the top consumer (rare, only during bulk ingestion):
    Terminal window
    docker exec innoqualis-backend python -c "from app.ai.vector_store import vector_store; vector_store.flush()"
  • Throttle PostgreSQL connections if db is the consumer — Phase 30.3 will add DBPoolExhausted to make this a first-class alert.

Resolution

Section titled “Resolution”
  • Confirmed leak: capture a memory profile before the next restart.
    Terminal window
    docker exec innoqualis-backend python -m memray run -o /tmp/mem.bin -p <pid> --duration 60
    docker cp innoqualis-backend:/tmp/mem.bin ./mem-$(date +%Y%m%d).bin
    Attach to the incident ticket. Common culprits: unbounded LRU caches, large pandas DataFrames held across requests, accidentally global request objects.
  • Bulk ingestion spike: not a leak — document in tasks/lessons.md, consider rate-limiting connector ingestion (Phase 23.6 has the hooks).

Prevention

Section titled “Prevention”
  • Per-container memory limit in docker/compose.yml — sets a hard ceiling that crashes the leaky worker into a fast restart instead of starving the host. Backend currently has no mem_limit; this is a Phase 30 follow-up.
  • Slow-growth alert: a 7-day rolling memory regression alert (memory grows >10% per hour, sustained) would catch slow leaks before they breach 85%. Not yet shipped; track as a follow-up.
  • memray baseline: run weekly memory profiles in CI to flag regressions per-PR. Phase 28 (test methodology) has the harness.

Last reviewed

Section titled “Last reviewed”

2026-05-29 — ops (initial runbook, shipped with Spec 30.8)