Runbook — DBPoolExhausted

Alert

DBPoolExhausted

Severity

critical

Audience

ops

Owner

Phase 30.3 / KAN-293

What this alert will do

A backend-emitted metric (db_pool_checkouts_waiting) > 0 sustained for 5 minutes. More precise than DatabaseConnectionIssues, which is a connection-error counter — this one catches pool exhaustion before requests start erroring.

Fallback procedure (until 30.3 lands)

# How many DB connections is the backend holding?
docker exec innoqualis-db psql -U postgres -c \
  "SELECT count(*), state FROM pg_stat_activity WHERE application_name LIKE 'innoqualis%' GROUP BY state;"

# Kill long idle-in-transaction sessions (the most common pool-leak signature):
docker exec innoqualis-db psql -U postgres -c \
  "SELECT pg_terminate_backend(pid) FROM pg_stat_activity \
   WHERE state='idle in transaction' AND state_change < now() - interval '5 minutes';"

# If the pool is genuinely exhausted under load, restart drains it cleanly:
docker compose -f docker/compose.yml restart backend

Common causes:

A code path that doesn’t release the SQLAlchemy session (forgotten db.close() in a background task).
A long-running query holding a transaction open (chase via pg_stat_activity).

Phase 30.3 spec
DatabaseConnectionIssues — broader DB alert
[ADR on connection pooling] — to be written with the rule

Last reviewed

2026-05-29 — ops (stub)