Runbook — PodNotReady

Alert

PodNotReady

Severity

warning

Audience

ops

Symptom

A container is running (not crashing — see PodCrashLoopBackOff for the crashing case) but its readiness probe has failed for 5+ minutes. The router / load balancer in front of it should be excluding it from rotation.

Impact

• If only one instance per service: same as BackendDown — see that runbook.
• If multiple instances: requests route to the healthy instances; capacity is reduced but the service is up.
• Customers may see intermittent latency if the failing instance is in rotation.

Diagnosis

1. docker ps --format 'table {{.Names}}\t{{.Status}}' — confirm the container is up but reporting unhealthy in the status column.
2. docker logs --tail 100 <container-name> — look for the underlying cause; often a downstream dependency (db, redis) the container can’t reach.
3. curl -fsS http://localhost:<port>/api/status for the backend, or curl -fsS http://localhost:3000/api/health for the frontend — manually hit the same endpoint the probe uses.

Mitigation

• If the dependency is down: fix that first (see DatabaseConnectionIssues, RedisConnectionIssues).
• If the container is genuinely stuck (no progress in logs): docker compose -f docker/compose.yml restart <service>.

Resolution

The underlying dependency outage is what needs fixing — PodNotReady is a symptom, not a cause. Chase the cause runbook (DB / Redis / Qdrant) that fired alongside this one.

Prevention

Tighten the readiness probe: it should fail fast and recover fast. A 5-minute window means we already lost capacity for 5 minutes before paging. Phase 30.3 will revisit probe tuning across all services.

Last reviewed

2026-05-29 — ops