Runbook — CertbotRenewalStalled / CertbotRenewalFailedRecently / PushgatewayDown

Note

Shipped in Phase 30.4 (KAN-294). Three rules in innoqualis.phase30.certbot route here:

• CertbotRenewalStalled — no successful renewal heartbeat in >24h (warning)
• CertbotRenewalFailedRecently — last attempt was a failure (critical)
• PushgatewayDown — pushgateway sidecar offline >10m (warning)

This runbook is the second line of defence against the 2026-05-17 incident; the first line is TLSCertExpiring30d and TLSCertExpiring7d on the blackbox SSL probe.

Alerts covered

Alert	Severity	Audience	Source
CertbotRenewalStalled	warning	ops	(time() - certbot_renewal_last_success_timestamp) > 86400
CertbotRenewalFailedRecently	critical	ops	last_failure_timestamp > last_success_timestamp
PushgatewayDown	warning	ops	up{job="pushgateway"} == 0

What this means

A bash script (infrastructure/monitoring/scripts/certbot-renewal-check.sh) runs:

1. Every 6 hours via cron (certbot-check.cron).
2. On every successful certbot renewal via --deploy-hook.

On each run it executes certbot renew --dry-run, then POSTs one of two metrics per domain to the Prometheus pushgateway:

certbot_renewal_last_success_timestamp{domain="hub.innoqualis.com"} <epoch>
certbot_renewal_last_failure_timestamp{domain="hub.innoqualis.com", reason="acme_challenge_failed"} <epoch>

The rules then fire on staleness or precedence between the two timestamps.

CertbotRenewalStalled is the heartbeat-missing warning — the script itself stopped running (cron broken, container down, network from script to pushgateway broken). CertbotRenewalFailedRecently is the precondition-precise alert — the script ran AND certbot’s renewal pipeline returned an error. PushgatewayDown is the meta-alert that catches the case where the carrier infrastructure for both above alerts has itself failed.

Diagnosis

CertbotRenewalStalled (warning)

# 1. Check whether the script is being invoked at all (cron health)
sudo journalctl -t certbot-renewal-check --since "-1d"

# 2. Manually invoke the script to see what fails
sudo /usr/local/bin/certbot-renewal-check.sh
echo "exit code: $?"
#   0 = renewed/dry-run OK
#   1 = certbot failed; failure metric pushed
#   2 = pushgateway unreachable
#   3 = certbot binary missing

# 3. Inspect the most recent certbot log
sudo tail -200 /tmp/certbot-renew.log

CertbotRenewalFailedRecently (critical)

# Inspect the labelled reason on the alert payload (in the Alertmanager email):
#   reason="acme_challenge_failed" → step 1 below (the 2026-05-17 cause)
#   reason="rate_limited"          → step 2
#   reason="connection_error"      → step 3
#   reason="dry_run_failed" (generic) → walk all three

# 1. ACME challenge path reachable? (2026-05-17 incident cause)
curl -fsS -o /dev/null -w '%{http_code}\n' \
  http://hub.innoqualis.com/.well-known/acme-challenge/test
# Expected: 404 (challenge file genuinely doesn't exist).
# 301 to HTTPS = nginx ACME block dropped — restore in nginx config.
# 502/503 = upstream broken before we reach the challenge handler.

# 2. Let's Encrypt rate limits hit?
# Inspect /tmp/certbot-renew.log for "too many" / "rate limit" lines.
# https://letsencrypt.org/docs/rate-limits/

# 3. Outbound connectivity to Let's Encrypt?
curl -fsS -o /dev/null -w '%{http_code}\n' https://acme-v02.api.letsencrypt.org/directory
# Expected: 200.

# 4. Inspect certbot's view of the cert lineage
sudo certbot certificates

# 5. Most recent renewal attempt
sudo docker logs --tail 500 innoqualis-certbot 2>&1 | grep -iE 'renew|error|fail' | tail -50

PushgatewayDown (warning)

# 1. Container status
sudo docker ps --filter name=eqms-pushgateway

# 2. Recent logs
sudo docker logs --tail 100 eqms-pushgateway

# 3. Direct health probe
curl -fsS http://localhost:9091/-/healthy

# 4. Restart (persistence volume preserves state)
cd /opt/innoqualis/monitoring && docker compose up -d pushgateway

Recovery

Symptom	Action
ACME challenge 301-redirects to HTTPS	Restore the location /.well-known/acme-challenge/ block in nginx port-80 server (2026-05-17 root cause)
Let’s Encrypt rate limit	Wait for the rate window (one week); do NOT force-renew
ACME directory unreachable	Check VPS outbound firewall + DNS resolution
Certbot binary missing / corrupt	apt install --reinstall certbot or rebuild the certbot container
Pushgateway crashed	docker compose up -d pushgateway (persistence preserves last-success timestamps)
Cron not running	systemctl status cron; check /etc/cron.d/certbot-renewal-check perms (must be 0644 and owned by root)

After remediation, force a renewal probe so the success timestamp advances immediately rather than waiting for the next 6h cron tick:

sudo /usr/local/bin/certbot-renewal-check.sh

Known false-positive scenarios

• Initial deployment / fresh certbot install — no success timestamp yet; CertbotRenewalFailedRecently cannot fire (its and last_success > 0 clause guards this), but CertbotRenewalStalled will fire once the 24h window expires unless the first successful probe runs. Mitigation: run the script manually as part of the deploy procedure to seed the metric.
• Pushgateway restart — without the persistence volume the last-success timestamp resets to zero on restart. Spec 30.4 configures persistence (--persistence.file=/data/persistence + pushgateway_data named volume) precisely to avoid this; if persistence is disabled (e.g. local dev), expect a 24h-after-restart false fire.
• Certbot lineage rename — if a cert is moved (e.g. hub.innoqualis.com lineage rotated to web.innoqualis.com) the old domain’s timestamp goes stale forever. Manually DELETE /metrics/job/certbot-renewal/instance/<old-domain> against the pushgateway to retire the series.

Escalation

1. Acknowledge in the control panel /alerts page (silences for 4h while you investigate).
2. If neither the cron nor a manual invocation can produce a fresh success timestamp within an hour, escalate to the on-call inbox (contact@innoqualis.com) and consider a manual certbot renew --force-renewal to extend the cert window while debugging.
3. If the 30.2 SSLCertNearExpiry rule has also fired on the same domain, the cert is now within 30 days — the situation is no longer purely cause-side and you have a hard deadline.

Compliance note

This alert + its successful runbook execution are evidence of “documented procedure for responding to system anomalies” per 21 CFR Part 11 §11.10(k). The pushgateway timestamp series provides the audit trail of renewal liveness over time.

Related

• Phase 30.4 spec
• Phase 30.2 spec — consequence-side defence
• tls-cert-expiring-30d — first warning, 30 days before expiry
• tls-cert-expiring-7d — hard-deadline warning
• 2026-05-17 incident post-mortem (in tasks/lessons.md)

Last reviewed

2026-05-29 — shipped with Spec 30.4 / KAN-294