Runbooks

Each alert defined in infrastructure/monitoring/prometheus/alert_rules.yml has a runbook page below. The Alertmanager email carries the runbook_url annotation; the control panel /alerts page renders it as a clickable link.

The CI check at backend/tests/test_alert_rules_have_runbooks.py asserts every rule has a runbook_url annotation and that the corresponding markdown file exists. A missing runbook fails the build — see the template for how to write one.

When docs.innoqualis.com itself is down (the worst case for runbook access), fall back to the source files under docs/starlight/src/content/docs/runbooks/ in the EQMS repo, or run make runbook ALERT=<name> once that helper lands (Phase 30.8 follow-up).

Critical-severity alerts

These page on-call immediately and (for audience: tenant or both) also email tenant admins.

Alert	Audience	Runbook
BackendDown / BackendDownSyntheticProbe	both / ops	Backend down (initial runbook covers both Spec 30.1 synthetic probe + 30.3 production SLO rule)
BackupJobFailed	ops	Backup job failed
CertbotRenewalStalled	ops	Certbot renewal stalled (stub — Phase 30.4)
DatabaseConnectionIssues	ops	Database connection errors
DBPoolExhausted	ops	DB pool exhausted (stub — Phase 30.3)
DeploymentFailed	ops	Deployment failed
DiskSpaceLow	ops	Disk space low
HighErrorRate	both	High 5xx error rate
HTTPSUptimeDown	both	HTTPS uptime down (stub — Phase 30.2)
InnoQualisBackendDown	both	InnoQualis backend down
InnoQualisFrontendDown	both	InnoQualis frontend down
ListmonkDown	ops	Listmonk down (stub — Phase 30.3)
N8nDown	ops	n8n down (stub — Phase 30.3)
PayloadCMSDown	ops	PayloadCMS down (stub — Phase 30.3)
PodCrashLoopBackOff	ops	Pod crash-loop backoff
QdrantDown	ops	Qdrant down (stub — Phase 30.3)
RedisConnectionIssues	ops	Redis connection errors
RedisDown	ops	Redis down (stub — Phase 30.3)
TLSCertExpiring7d	ops	TLS cert expiring within 7 days (stub — Phase 30.2)
TwentyDown	ops	Twenty CRM down (stub — Phase 30.3)

Warning-severity alerts

These surface in the control panel and email ops; they do not escalate to tenant admins (per the audience-severity gate in ADR 0033).

Alert	Audience	Runbook
BackendP95Regression	ops	Backend p95 regression (stub — Phase 30.3)
Backend5xxRateSpike	ops	Backend 5xx rate spike (stub — Phase 30.3)
CertificateExpiringSoon	ops	Certificate expiring soon
HighCPUUsage	ops	High CPU usage
HighMemoryUsage	ops	High memory usage
HighResponseTime	ops	High response time
InnoQualisHighCPUUsage	ops	InnoQualis backend CPU
InnoQualisHighMemoryUsage	ops	InnoQualis backend memory
PodNotReady	ops	Pod not ready
TLSCertExpiring30d	ops	TLS cert expiring within 30 days (stub — Phase 30.2)

Related

• On-call rotation and escalation — who responds, when, and how to escalate.
• Incident response workflow — acknowledge / silence / postmortem cadence.
• Phase 30 overview — the alert routing / severity / audience model.
• ADR 0033 — Alert routing topology and severity model
• Runbook template — copy this when adding a new alert.