Runbook — HTTPSSlow

Note

Initial stub. Phase 30.2 (KAN-292) ships the HTTPSSlow alert rule. The runbook body below covers the diagnosis steps that apply to every domain in the probe inventory; the next operational rotation will tighten it with per-domain triage notes.

Alert

HTTPSSlow

Severity

warning

Audience

ops

Owner

Phase 30.2 / KAN-292

What this alert means

The blackbox-exporter probe against https://{domain}/ has measured a p95 latency above 2 seconds for more than 10 minutes (histogram_quantile(0.95, rate(probe_duration_seconds_bucket{job="blackbox-https-2xx"}[10m])) > 2).

It is the latency-side complement to HTTPSDown — it catches the class of degradation where “everything still responds, but slowly”, which the binary up/down probe misses entirely.

audience: ops because slow-but-up is not a customer-data-flow incident — tenants do not need an email until it crosses into outage territory (which HTTPSDown covers separately).

Diagnosis

1. Identify the affected domain from {{ $labels.domain }} in the alert payload.
2. Reproduce the latency from outside the VPS:

   for i in 1 2 3 4 5; do
     curl -fsS -o /dev/null -w 'attempt %{http_code} total %{time_total}s connect %{time_connect}s ssl %{time_appconnect}s\n' \
       "https://<domain>/" --max-time 15
   done

3. Check upstream backend latency in the Grafana “EQMS Overview” dashboard — if HighResponseTime is also firing, the root cause is the backend, not the edge.
4. Check the certbot / nginx logs on the VPS for recent reloads that might have invalidated upstream connection pools.
5. Check the cAdvisor container metrics for CPU saturation on the innoqualis-backend or innoqualis-frontend container.

Recovery

There is no single recovery action — HTTPSSlow is a symptom, not a cause. Pursue the diagnosis tree above; common recoveries:

• Backend CPU saturation → scale workers (docker compose up -d --scale backend=N)
• DB pool exhausted → see DatabaseConnectionIssues
• CDN cache thrash → check Cloudflare / nginx cache hit ratio

Known false-positives

• During scheduled cert rotations (~hourly window when certbot reloads nginx), p95 latency can briefly spike. The for: 10m window absorbs short reloads; a sustained breach is real.

Related

• HTTPSDown — binary up/down sibling
• HighResponseTime — backend-side latency rule
• Phase 30.2 spec

Last reviewed

2026-05-29 — ops (initial)