Skip to content
InnoQualis InnoQualis Docs

Runbook — DatabaseConnectionIssues

Alert

Section titled “Alert”

DatabaseConnectionIssues

Severity

Section titled “Severity”

critical

Audience

Section titled “Audience”

ops

Symptom

Section titled “Symptom”

increase(database_connection_errors_total[5m]) > 10 — the backend is logging connection errors against PostgreSQL faster than 10 / 5min. In the backend logs, lines like psycopg2.OperationalError: connection to server failed flood in.

Impact

Section titled “Impact”
  • All write operations (and many reads) fail.
  • Customers see “Something went wrong” everywhere.
  • Likely chains to HighErrorRate firing alongside.

Diagnosis

Section titled “Diagnosis”

  1. Is the DB container running?

    Terminal window
    docker ps --filter name=innoqualis-db --format 'table {{.Names}}\t{{.Status}}'

    If not running → start it; that’s the fix.

  2. Can we connect at all?

    Terminal window
    docker exec innoqualis-db psql -U postgres -c 'SELECT 1;'

    If this hangs or fails → PostgreSQL is in recovery / out of connections / out of disk.

  3. Connection count vs. limit.

    Terminal window
    docker exec innoqualis-db psql -U postgres -c "SELECT count(*) FROM pg_stat_activity;"
    docker exec innoqualis-db psql -U postgres -c "SHOW max_connections;"

    If the first is close to the second → pool exhaustion. See Mitigation.

Mitigation

Section titled “Mitigation”
  • DB down → docker compose -f docker/compose.yml up -d db, then docker compose restart backend to re-establish pools.
  • Connection exhaustion → kill idle: docker exec innoqualis-db psql -U postgres -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state='idle' AND query_start < now() - interval '10 minutes';"
  • Disk full → see DiskSpaceLow runbook.

Resolution

Section titled “Resolution”
  • Persistent pool exhaustion → backend is leaking connections. Audit Depends(get_db) usage; common bug is forgetting db.close() in non-FastAPI contexts (background tasks, scripts).
  • DB crash → check docker logs innoqualis-db for the cause. If OOM, increase memory; if disk corruption, restore from backup.

Prevention

Section titled “Prevention”
  • Connection pool ceiling tuned to max_connections - 20 (reserve headroom for psql sessions).
  • DBPoolExhausted alert (Phase 30.3) — more specific than this rule; fires before DatabaseConnectionIssues.
  • Monthly recovery drill: simulate DB crash + restore from backup.

Last reviewed

Section titled “Last reviewed”

2026-05-29 — ops