Skip to content
InnoQualis InnoQualis Docs

Runbook — PodCrashLoopBackOff

Alert

Section titled “Alert”

PodCrashLoopBackOff

Severity

Section titled “Severity”

critical

Audience

Section titled “Audience”

ops

Symptom

Section titled “Symptom”

A container (or, in the Kubernetes-naming legacy of this alert, “pod”) has been in a crash → restart → crash loop for more than 2 minutes. The alert labels carry pod and namespace; on the current docker-compose stack these map to the docker compose service name.

In the control panel /alerts page the alert is red. Customers see 502 / 503 from whichever service the crashing container backs (usually the FastAPI backend).

Impact

Section titled “Impact”
  • Critical because the service is effectively down — restart-loop = no working instance to serve traffic.
  • All tenants affected if the backend or frontend container is the one crashing.
  • Data writes in flight when the crash started may have been lost (in-memory state, queued background jobs not yet persisted).

Diagnosis

Section titled “Diagnosis”

  1. Which service is crashing and why?

    Terminal window
    docker ps -a --filter "status=restarting" --format 'table {{.Names}}\t{{.Status}}\t{{.CreatedAt}}'

    Identifies the container. Status will show Restarting (N) X seconds ago.

  2. What’s the crash signal?

    Terminal window
    docker logs --tail 200 <container-name> 2>&1 | tail -100

    The crash reason is almost always in the last 50 lines. Common signatures:

    • ImportError: cannot import name 'X' from 'app.Y' → bad code in a recent deploy
    • psycopg2.OperationalError: connection to server failed → DB unreachable from this container
    • pydantic.ValidationError: ... is not a valid environment variable → missing / malformed env var
    • MemoryError or no log output at all → OOM kill (check dmesg | grep -i oom)
    • alembic.util.exc.CommandError → migration failed; backend exits before serving (see backend/start.sh)

  3. What changed recently?

    Terminal window
    git -C /opt/innoqualis log --oneline --since='6 hours ago' | head -10
    docker inspect <container-name> --format '{{.Config.Image}} {{.Created}}'

    Correlate the container creation timestamp with recent git commits — almost always the most recent commit / deploy.

Mitigation

Section titled “Mitigation”

In order of preference:

  1. If a recent deploy broke it — roll back:

    Terminal window
    git -C /opt/innoqualis revert HEAD --no-edit
    make prod-up

    Fastest recovery. Apologise to whoever wrote the offending commit later.

  2. If migration failed — bypass and run manually:

    Terminal window
    docker exec innoqualis-db psql -U postgres -d eqms -c "SELECT version_num FROM alembic_version;"
    # then in a fresh shell:
    docker compose -f docker/compose.yml run --rm backend python -m alembic upgrade head

    start.sh runs alembic upgrade head on boot and exits 1 on failure — that’s the loop. Once migration succeeds manually, the loop clears.

  3. If a missing env var — patch and restart: Edit docker/.env, then docker compose -f docker/compose.yml up -d --force-recreate <service>.

  4. If OOM kill — chain to HighMemoryUsage runbook for the root cause.

Resolution

Section titled “Resolution”
  • Bad deploy: keep the rollback live. Open a ticket for the offending commit. Run /sanity-check on the fix before re-deploying.
  • Migration failure: don’t just bypass — investigate why migration failed (e.g. a column drop on data that doesn’t satisfy the schema). Fix the migration itself, then re-deploy.
  • Missing env var: add the var to docker/.env.example and document in docs/starlight/src/content/docs/getting-started/profile-and-settings.md so it doesn’t recur.
  • OOM: cap container memory in docker/compose.yml so it’s killed fast and cleanly. Profile the leak per the HighMemoryUsage runbook Resolution section.

Prevention

Section titled “Prevention”
  • Health-check on every service in docker/compose.yml — currently inconsistent across services; if a container takes 30s to crash, docker thinks it’s healthy until then. Standardise to healthcheck.start_period: 30s and interval: 10s.
  • Smoke deploy in staging first — Phase 13.5.1 (tenant isolation closeout) added the staging pipeline; if a deploy went straight to prod and crashed, the staging skip is a process bug.
  • Migration dry-run in CI — alembic upgrade head against a copy of prod, gated on every PR that touches backend/alembic/versions/. Tracked in Phase 28 (test methodology).

Last reviewed

Section titled “Last reviewed”

2026-05-29 — ops (initial runbook, shipped with Spec 30.8)