Incident response workflow

Lifecycle

1. Alert fires → Alertmanager routes to contact@innoqualis.com (and tenant admins, if audience: tenant or both and severity: critical).
2. Acknowledge within the response target.
3. Diagnose using the alert’s runbook_url annotation.
4. Mitigate — get the customer-visible impact back to zero. Mitigation is not the same as resolution.
5. Resolve — close out the underlying cause. Often takes longer than mitigation.
6. Review — write up what happened, why, and what changes are needed.

Acknowledging

See the on-call doc for the click-path. Two principles:

• Acknowledge ≠ silence. Acknowledging means “I see this and I’m working it”; the alert keeps re-firing if the condition persists, so other team members can still see it’s active.
• Comment your hypothesis. Even “investigating, suspect recent deploy” is enough — gives the next responder context if you have to hand off.

Diagnosing

The runbook_url annotation on every alert links to the relevant runbook under /runbooks/. Each runbook has a Diagnosis section with the first three commands to run. Run them in order.

If the diagnosis exhausts the runbook and the root cause is still unclear:

1. Check Sentry (once Phase 28.2 lands) for correlated exceptions.
2. Check Grafana dashboards for the affected service.
3. Check tasks/lessons.md for prior incidents with similar symptoms.
4. Check the most recent 24h of commits — most incidents are caused by recent changes.

Mitigating vs. resolving

These are distinct, and the order matters.

Step	Goal	Time horizon
Mitigate	Customer impact is gone. Service is up, even if degraded.	Minutes to hours.
Resolve	Root cause is fixed in code / config. Won’t recur.	Hours to days.

A typical pattern: mitigate by rolling back, resolve by fixing-forward in a PR with a regression test under backend/tests/regressions/ or frontend/e2e-tests/regressions/.

Silencing

Use Alertmanager silences sparingly. The intent is for known false-positives or during planned maintenance — not “this alert is annoying”.

Every silence:

• Must have a comment with the silence reason.
• Must have a max duration of 24 hours (longer requires re-justification).
• Is logged to the incident_log table (Phase 30.5) for compliance review.

If an alert is genuinely noisy / false-positive in steady state, don’t silence — change the rule. File a Phase 30 follow-up PR.

Post-incident review

Required for every critical incident. Write within 5 working days.

Template

# Incident: <short title>

**Date**: YYYY-MM-DD
**Duration**: HH:MM mitigation, HH:MM resolution
**Severity**: critical / warning
**Audience**: ops / tenant / both
**Alert**: AlertName
**Affected tenants**: [list, or "platform-wide"]
**On-call responder**: <name>

## Summary

One paragraph: what broke, what customers saw, what we did, what we learned.

## Timeline

- HH:MM — Alert fires
- HH:MM — Ack
- HH:MM — First diagnosis hypothesis (X)
- HH:MM — Mitigation deployed (Y)
- HH:MM — Customer impact returns to zero
- HH:MM — Root cause identified (Z)
- HH:MM — Resolution shipped

## Root cause

One paragraph. Be precise. "Recent deploy broke things" is not a root cause; "commit abc1234 introduced X which interacts with Y to produce Z" is.

## Customer impact

Quantify: # of tenants affected, # of failed requests, duration of degraded behaviour.

## What worked

What about our process / tooling helped us get through this faster?

## What didn't

What slowed us down? Be honest. Common culprits: stale runbook, missing alert, undocumented config.

## Action items

Tracked in Jira (KAN-xxx) and tasks/lessons.md:

- [ ] **<title>** — what / who / when
- [ ] ...

Where to file the writeup

1. Add an entry to tasks/lessons.md with the same date.
2. Create the action items as Jira tickets linked to the lessons.md entry.
3. If the incident exposed a missing alert: file a follow-up to add it. If the incident exposed a stale runbook: update the runbook in the same PR as the fix so the action item is closed automatically.

Compliance evidence

The combination of:

• Alert fires logged (Alertmanager + incident_log table from Phase 30.5)
• Acknowledgement logged (with user, timestamp, comment)
• Runbook procedures executed (the runbook_url chain)
• Post-incident review written (this section)

…together satisfies the “documented procedures and controls for responding to system anomalies” requirement under 21 CFR Part 11 §11.10(k). The runbooks and this workflow are the artefact regulators (and customer auditors) will ask about.

Related

• On-call rotation — who responds and when.
• Runbook index — one page per alert.
• Phase 30 overview
• tasks/lessons.md — the existing project-wide lessons-learned log.

Last reviewed

2026-05-29 — founder