SRE Runbooks
These runbooks turn FDAI’s SRE contracts into repeatable operator procedures. Upstream documents the required safety checks, evidence, decisions, and terminal outcomes. A downstream fork supplies environment-specific commands, resource names, owners, paging integrations, and rollback implementations.
Incident operations
Section titled “Incident operations”| Procedure | Use it when |
|---|---|
| Incident triage | A new incident needs scope, severity, ownership, and investigation |
| SLO burn response | Multi-window error-budget burn raises a detected issue |
| RCA evidence collection | An investigation needs a bounded, cited evidence set |
| Incident mitigation and rollback | A response plan proposes a governed change |
| Postmortem workflow | A resolved incident needs review and follow-up |
Preparedness
Section titled “Preparedness”| Procedure | Use it when |
|---|---|
| Deployment recovery | A protected plan, offline kit, or startup-readiness check blocks safe deployment |
| Deep DB-DR restore drill | PostgreSQL restore evidence must be refreshed |
| Control-plane regional failover and failback | A regional outage or scheduled full control-plane recovery drill is declared |
| Chaos game day | A promoted fault scenario is exercised |
| Alert tuning | Noise, misses, or stale routing need measured correction |
| Alert noise governance | Observation-first alert assessment or one of the four exact-plan manual-PR actions needs prerequisite, verification, and recovery guidance |
Governance and setup
Section titled “Governance and setup”Required runbook contract
Section titled “Required runbook contract”Every executable procedure defines owner and approver, bounded scope, preflight, stop conditions, rollback, evidence, audit reference, and terminal no-op behavior. If any required item is unavailable, stop and route to review.