Skip to content
InnoQualis InnoQualis Docs

Runbook — CertificateExpiringSoon

Alert

Section titled “Alert”

CertificateExpiringSoon

Severity

Section titled “Severity”

warning

Audience

Section titled “Audience”

ops

Symptom

Section titled “Symptom”

(ssl_cert_not_after - time()) / 86400 < 30 — at least one TLS certificate has fewer than 30 days remaining. The alert’s instance label names the host whose cert is expiring.

Impact

Section titled “Impact”
  • Today: none.
  • In 30 days: customers see ERR_CERT_DATE_INVALID — full outage. Detection-to-impact ratio: 30 days, hence why this is warning not critical.

This alert is the safeguard against the 2026-05-17 incident (hub.innoqualis.com expired silently because the ACME challenge was misconfigured). Phase 30.2 adds finer-grained TLSCertExpiring7d / TLSCertExpiring30d rules with the audience-aware tagging the broader Phase 30 model needs.

Diagnosis

Section titled “Diagnosis”

  1. Which cert and which domain?

    Terminal window
    echo | openssl s_client -connect <domain>:443 -servername <domain> 2>/dev/null | openssl x509 -noout -dates -subject

  2. Is certbot configured to renew it?

    Terminal window
    docker exec innoqualis-certbot certbot certificates

    Each cert should show “Renewal: enabled” and a domains: entry matching the alert.

  3. Has renewal been attempted?

    Terminal window
    docker logs --tail 200 innoqualis-certbot 2>&1 | grep -E 'renew|error|fail' | tail -20

    Look for ACME challenge failures — the exact failure mode of the 2026-05-17 incident.

Mitigation

Section titled “Mitigation”

  • Force renew now:

    Terminal window
    docker exec innoqualis-certbot certbot renew --force-renewal --cert-name <domain>

    Followed by:

    Terminal window
    docker exec innoqualis-nginx nginx -s reload

  • If ACME challenge fails: see the CertbotRenewalStalled stub runbook (Phase 30.4 fills the body). Quick check: curl -fsS http://<domain>/.well-known/acme-challenge/test should return 404, NOT 301-redirect to HTTPS.

Resolution

Section titled “Resolution”
  • Once renewed, the cert should auto-renew every ~60 days. Verify cron / systemd timer for certbot is active.
  • Document the renewal in tasks/lessons.md if the renewal failure was a recurrence of the 2026-05-17 incident.

Prevention

Section titled “Prevention”
  • Phase 30.2 adds finer-grained alerts: TLSCertExpiring30d (warning) and TLSCertExpiring7d (critical).
  • Phase 30.4 adds CertbotRenewalStalled — alerts if renewal hasn’t logged success in 14 days, catching silent failures within 24h.
  • Cron-monitored renewal command (already partial; needs verifying after Phase 30.4).

Last reviewed

Section titled “Last reviewed”

2026-05-29 — ops