Skip to content
InnoQualis InnoQualis Docs

Runbook — InnoQualisHighMemoryUsage

Alert

Section titled “Alert”

InnoQualisHighMemoryUsage

Severity

Section titled “Severity”

warning

Audience

Section titled “Audience”

ops

Symptom

Section titled “Symptom”

process_resident_memory_bytes{job="innoqualis-backend"} > 1073741824 for 5 minutes — at least one backend worker is over 1 GB resident.

Impact

Section titled “Impact”
  • Risk of progression to HighMemoryUsage (node level) and then OOM kills.
  • Container memory ceiling is currently uncapped — there’s no graceful fast-restart.

Diagnosis

Section titled “Diagnosis”

Run the HighMemoryUsage runbook diagnosis steps — same machinery, different scope.

In addition:

  1. Per-worker RSS:

    Terminal window
    docker exec innoqualis-backend ps -o pid,rss,cmd -C python

  2. Recent endpoint pattern:

    Terminal window
    docker logs --tail 1000 innoqualis-backend 2>&1 | grep -E '"path"' | jq -r .path | sort | uniq -c | sort -rn | head

    A single endpoint dominating may be the leak source.

Mitigation

Section titled “Mitigation”
  • Restart the backend: docker compose -f docker/compose.yml restart backend (graceful drain; ~30s).

Resolution

Section titled “Resolution”

See the HighMemoryUsage Resolution section — profiling with memray, identifying the leak, fix.

Prevention

Section titled “Prevention”
  • Per-container mem_limit in docker/compose.yml (e.g. 1.5 GB hard ceiling). Currently absent.
  • Memory regression test in CI.

Last reviewed

Section titled “Last reviewed”

2026-05-29 — ops