Runbook — DiskSpaceLow

Alert

DiskSpaceLow

Severity

critical

Audience

ops

Symptom

A filesystem on a node has exceeded 90% utilisation for more than 5 minutes. The instance + mountpoint labels on the alert name the host and filesystem. Grafana’s “Node Filesystem” panel shows the usage curve.

If the data volume hits 100%, PostgreSQL refuses new writes (ERROR: could not extend file), uploads fail with 500s, and the backend logs flood with disk-full errors.

Impact

Critical because cliff is close: at 90% there are usually only hours (not days) of headroom on a busy node.
Document uploads will fail when the volume is full.
PostgreSQL writes will fail when its data volume is full — full outage for all tenants.
Backups may fail if there’s no room for the dump file.

Diagnosis

Which filesystem is full and by how much?
Terminal window
```
ssh ops@<instance> 'df -h | sort -k5 -hr | head -10'
```
The Use% column ranked highest — usually /var/lib/docker or a mounted data volume.
What’s eating the space on that filesystem?
Terminal window
```
ssh ops@<instance> 'du -h --max-depth=2 /var/lib/docker 2>/dev/null | sort -hr | head -20'
```
Common offenders (in order of frequency):
- volumes/innoqualis_db_data — PostgreSQL WAL not being archived/recycled
- volumes/innoqualis_documents — accumulated uploads
- containers/<id>/<id>-json.log — unrotated container logs
- overlay2/<sha> — stale image layers from old deploys
For the heaviest container — is it logs?
Terminal window
```
ssh ops@<instance> 'ls -lh /var/lib/docker/containers/*/*.log | sort -k5 -hr | head -5'
```
A single *.log file >1 GB means the container’s been running without log rotation for a while.

Mitigation

Pick by what diagnosis showed. The first two are SAFE; the third needs care.

Container logs (safe):
Terminal window
```
ssh ops@<instance> 'for f in /var/lib/docker/containers/*/*.log; do truncate -s 0 "$f"; done'
```
Truncates in place — keeps the container running, frees space immediately. Schedule a follow-up to set logging.options.max-size in compose.
Stale docker images (safe):
Terminal window
```
ssh ops@<instance> 'docker image prune -a -f --filter "until=168h"'
```
Removes images not used by any running container, older than 7 days.
PostgreSQL WAL (requires care — DO NOT delete WAL files manually):
Terminal window
```
docker exec innoqualis-db psql -U postgres -c 'CHECKPOINT;'
docker exec innoqualis-db psql -U postgres -c "SELECT pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), '0/0'));"
```
If WAL is the problem, the cause is almost always a stuck replication slot (SELECT * FROM pg_replication_slots WHERE active = false;) — drop the inactive slot and WAL recycles within minutes.

Resolution

Logs were the problem: set logging in docker/compose.yml per-service:
```
logging:
  driver: json-file
  options:
    max-size: "100m"
    max-file: "3"
```
This is a one-line fix per service, ship in the next deploy.
Document storage: if user uploads are filling disk, this is a Phase 26 (cloud migration → S3) priority. Pre-cloud: increase the data volume size (Hetzner panel resize → resize2fs on the host).
PostgreSQL WAL: if a stale replication slot was the cause, audit how it was created and document. If WAL is genuinely growing because of write volume, plan PITR backup tuning + volume resize.

Prevention

Two-tier alert: warn at 75%, critical at 90%. Current rule only fires at 90% — too close to the cliff. File a follow-up to add DiskSpaceWarn at 75%.
Per-service log size caps in compose (see Resolution). One-time change, eliminates the most common cause of this alert.
Cloud migration (Phase 26): moves document storage off local disk to object storage. Eliminates the second most common cause.

Last reviewed

2026-05-29 — ops (initial runbook, shipped with Spec 30.8)