Skip to content
InnoQualis InnoQualis Docs

Runbook — HighResponseTime

Alert

Section titled “Alert”

HighResponseTime

Severity

Section titled “Severity”

warning

Audience

Section titled “Audience”

ops

Symptom

Section titled “Symptom”

histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 2 — the 95th-percentile response time is over 2 seconds. Customers experience the app as sluggish; rarely outright failures, but enough latency to harm flow.

Impact

Section titled “Impact”
  • Latency visible to all tenants.
  • Document upload, AI queries, and dashboard loads all feel slow.
  • Risk of progression to HighErrorRate if the slowness causes downstream timeouts.

Diagnosis

Section titled “Diagnosis”

  1. Which endpoint is slow?

    Terminal window
    docker logs --tail 1000 innoqualis-backend 2>&1 | grep -E 'duration_ms":[0-9]{4,}' | jq -r '.path + " " + (.duration_ms|tostring)' | sort | uniq -c | sort -rn | head -10

  2. Is the DB slow?

    Terminal window
    docker exec innoqualis-db psql -U postgres -d eqms -c "SELECT pid, now()-query_start AS duration, query FROM pg_stat_activity WHERE state='active' ORDER BY duration DESC LIMIT 5;"

  3. Is Qdrant slow? (AI / embedding endpoints)

    Terminal window
    curl -fsS http://localhost:6333/cluster | jq

Mitigation

Section titled “Mitigation”
  • Restart backend if a worker is stuck (docker compose -f docker/compose.yml restart backend).
  • Kill long-running queries: docker exec innoqualis-db psql -U postgres -c "SELECT pg_cancel_backend(<pid>);"
  • Disable the slow feature if it’s an AI endpoint (feature flag in frontend/lib/config.ts).

Resolution

Section titled “Resolution”
  • Slow query → add an index, paste EXPLAIN ANALYZE into the fix PR.
  • AI stall → check OpenAI / embedding provider status; consider failover.
  • Capacity → see HighCPUUsage runbook for the scale-up path.

Prevention

Section titled “Prevention”
  • Per-endpoint p95 budget gate in CI (Phase 28 test methodology).
  • pg_stat_statements enabled + weekly slow-query review.
  • Sentry performance traces (Phase 28.2).

Last reviewed

Section titled “Last reviewed”

2026-05-29 — ops