On-call rotation and escalation

Posture (today)

InnoQualis runs solo on-call. The founder owns every alert until a second engineer is onboarded into the rotation (planned post-Phase 26 cloud migration).

Note

Solo on-call is a deliberate, time-boxed posture: the volume of production traffic and the maturity of the automated runbooks make it sustainable. Phase 30’s runbooks (this site) are the load-bearing artefact that makes it sustainable — every alert links to a documented procedure so the same person being paged isn’t also the only person who knows what to do.

Rotation owner

Role	Person	Channels
Primary on-call	Founder (Bruno Boto)	Email contact@innoqualis.com, phone (in tasks/contacts.md, ops-only)
Backup	— (none yet — see below)	—

The contact@innoqualis.com inbox is the single entry point. Every alert from Alertmanager (Phase 30.1) routes here. The inbox is monitored on all working days (Mon–Fri, 09:00–18:00 Europe/Lisbon) and best-effort outside those hours.

Severity to response-time matrix

Severity	Audience	Target ack	Target resolution	Notes
critical	ops	15 min (in-hours) / 60 min (out-of-hours)	2 hours	Page immediately; runbook first
critical	tenant	15 min	1 hour	Tenant admins also notified — keep them informed via the in-app status page
critical	both	15 min	2 hours	Most severe — multi-tenant outage
warning	any	1 hour (in-hours only)	next business day	Triage, do not page; investigate root cause
info	any	end of day	next sprint	Aggregated; review weekly

These are targets, not SLAs. Public SLAs are out of scope until Phase 29 (go-live).

Escalation matrix

When the primary cannot respond:

1. First 30 minutes — the alert sits in the contact@innoqualis.com inbox. Alertmanager grouping + the 5-minute dedup window means re-fires don’t flood.
2. 30-60 minutes — if no acknowledgement, Alertmanager re-pages (repeat_interval: 30m in Phase 30.1’s config).
3. 60+ minutes (critical only) — if the alert is audience: tenant or both, tenant admins have already been notified directly. Communicate via the public status page that the issue is acknowledged but resolution is delayed.
4. 24+ hours unacknowledged — escalate to the legal / compliance contact via compliance@innoqualis.com (defined in tasks/contacts.md). For regulated tenants, sustained outage may trigger a separate compliance reporting path.

There is no engineering escalation tier today (solo on-call). When the second engineer joins, this section gets a “secondary on-call” row.

Acknowledging an alert

1. Open the control panel at https://control.innoqualis.com/alerts (Phase 30.5 ships this page).
2. Click Acknowledge on the firing alert — creates an Alertmanager silence and writes an entry to the incident_log audit table.
3. Comment with your action plan in the alert detail panel. The comment is appended to the alert’s history and emailed to contact@innoqualis.com so the team has visibility.
4. Open the runbook linked from the alert (runbook_url annotation) and follow it.

Acknowledgement does not silence Alertmanager re-fires for the underlying condition — only an explicit silence (Alertmanager UI) does that. The intent is: “I’m working on it” should not mean “stop telling me about it”.

After hours

The product targets ISO 9001/13485 and 21 CFR Part 11 regulated industries. No customer’s QMS depends on 03:00 incident response from our side — their internal procedures handle the immediate situation; we restore the platform during the next working window. This is documented in customer onboarding.

That said: critical audience: tenant alerts (genuine tenant-blocking outages) DO page out-of-hours best-effort, because the regulated-industry context means a multi-day outage is unacceptable.

Post-incident review

Every critical incident gets a written review within 5 working days. Template lives at /operations/incident-response/#post-incident-review. The review is filed in tasks/lessons.md per the project’s existing post-incident pattern.

Deferred

Out of scope until Phase 26 (cloud migration) and Phase 29 (go-live):

• Paid on-call tooling (PagerDuty, Opsgenie, BetterStack) — the cost is not justified pre-revenue. Once paid customers depend on the rotation, the tooling moves in.
• Multi-tier escalation with secondary / tertiary on-call engineers — waits on team growth.
• Public SLA documentation — Phase 29’s task, includes the customer-facing version of the response-time matrix above.
• Disaster recovery / business continuity plan — separate workstream, out of scope for Phase 30.

Related

• Incident response workflow — the step-by-step for actually working an incident.
• Runbook index — one page per alert; the operator’s working reference.
• Phase 30 overview
• ADR 0033 — Alert routing and severity model

Last reviewed

2026-05-29 — founder