Measuring SRE Outcomes
FDAI measures SRE improvement through outcomes and guard metrics, not through the percentage of decisions made automatically. Every comparison uses the same scenario set, a stated measurement window, and paired baseline and treatment evidence.
Outcome metrics
Section titled “Outcome metrics”| Metric | What it answers |
|---|---|
| MTTR distribution | How long resolution takes at the mean, median, and p90 |
| Auto-resolution rate | Which events reached the correct terminal outcome with no human touchpoint and no later rollback |
| Human touchpoints | How much operator work remains per incident |
| Change lead time | How long a governed change takes from request to merge |
| Cost per resolved event | What attributable platform and inference spend each result consumed |
Guard metrics
Section titled “Guard metrics”Track change-failure rate, false positives, false negatives, rollback rate, policy-violation escapes, audit gaps, verifier failures, and mixed-model disagreement. An outcome improvement does not count if a guard metric regresses past its threshold.
Measurement contract
Section titled “Measurement contract”- Freeze the scenario-set version and input distribution.
- Record the baseline model, rules, thresholds, adapters, and catalog versions.
- Run treatment against the same scenarios and observation window.
- Report sample size, missing data, confidence, and distribution, not only an average.
- Keep observation mode and enforce outcomes separate.
- Demote a capability when measured guard metrics regress.
Avoid misleading claims
Section titled “Avoid misleading claims”- Do not claim a multiplier without paired measurements.
- Do not treat a missing projection as zero.
- Do not merge mean and p90 into one latency statement.
- Do not count a later rollback as successful auto-resolution.
- Do not compare different scenario sets without labeling the change.
Next steps
Section titled “Next steps”| To learn about | Read |
|---|---|
| Canonical formulas and windows | Goals and metrics |
| The named scenario sets and evidence levels | Scenario validation inventory |
| How SLO burn measures workload impact | SLOs and error budgets |
| How observation mode evidence controls promotion | Observe, then enable changes |
| How audit evidence is reconstructed | Read the audit log |