Runbook — RedisConnectionIssues

Alert

RedisConnectionIssues

Severity

critical

Audience

ops

Symptom

redis_up == 0 for 2+ minutes. Redis is used for: session token storage, the LLM-budget Redis cache (Spec 24.8), background job queues, and the recipient resolver cache (Phase 30.6). When it’s down:

• Users may stay logged in (JWT in localStorage works without Redis) but operations that touch sessions go slow.
• LLM-budget enforcement falls back to DB-only path (slow, but correct).
• Background workers stall.

Impact

• Degraded experience for all tenants; not a full outage as long as the JWT path keeps working.
• Background jobs (training reminders, notifications) queue locally in workers and replay when Redis returns.
• LLM cost-tracking falls back to DB lookup; ~5× latency on AI endpoints.

Diagnosis

1. Is the Redis container running?

   docker ps --filter name=innoqualis-redis --format 'table {{.Names}}\t{{.Status}}'

2. Can we PING it?

   docker exec innoqualis-redis redis-cli PING

   Expected: PONG. Anything else = bad.

3. Memory pressure? (Redis evicts under pressure)

   docker exec innoqualis-redis redis-cli INFO memory | grep -E 'used_memory_human|maxmemory_human|evicted_keys'

Mitigation

• Container down: docker compose -f docker/compose.yml up -d redis
• Memory full: docker exec innoqualis-redis redis-cli FLUSHDB — destroys cache, but cache is rebuildable. Avoid FLUSHALL.
• Network: docker network inspect innoqualis-network — Redis on the wrong network is the most common cause of post-deploy Redis errors.

Resolution

• Tune maxmemory + maxmemory-policy allkeys-lru so Redis evicts cleanly instead of erroring out.
• Move to a managed Redis (ElastiCache / Memorystore) in Phase 26 cloud migration.

Prevention

• Memory ceiling alert at 80% of maxmemory.
• Graceful degradation tests: every Redis-consuming code path MUST work (slowly) when Redis is unreachable. Spec 24.8 already does this for LLM budget; audit others.

Last reviewed

2026-05-29 — ops