Goals and Metrics

The roadmap optimizes for autonomy with proof. Every autonomy claim is backed by a measured baseline; nothing is asserted from estimation. Improvement factors below (5×, large reduction, 1/5) are targets, not achieved results - they may only be stated as achieved once both the reference baseline and the FDAI treatment have been measured on the same scenario set (see Measurement-First Rule).

This document is the source of truth for KPIs. It aligns with the tier coverage targets in architecture.instructions.md and is operationalized by phase-0-instrumentation.md.

Implementation status

Implementation scope

Area	State	Evidence	Notes
Deterministic KPI and guard-metric aggregation	implemented	core/measurement/mttr.py; dora.py; regression.py; focused tests under tests/core/measurement/	MTTR, change, regression, latency, model, and pattern metrics have executable reducers and fail-closed checks.
Promotion and operational evidence evaluation	implemented	core/measurement/promotion_gate.py; operational_promotion.py; focused promotion tests	Promotion evaluation binds revision, scenario, samples, confidence, guards, and outcome evidence. It does not itself prove a live cohort exists.
Frozen scenario-set accounting	in-progress	tests/scenarios/manifests/v2026.07.json; test_frozen.py; test_v2026_07_replay.py	The manifest and executable coverage checks exist, but SRE and other required constitutional dimensions remain incomplete as described below.
Live KPI baseline, treatment, and dashboard closure	in-progress	Data Collection and Telemetry; config/constitution-traceability.json requirement FDAI-CONST-002	Runtime records and jobs exist, but no retained full live baseline/treatment cohort proves all success and zero-threshold guard metrics on one pinned revision.

Implementation history

Date	State	Change	Evidence	Remaining
2026-08-19	implemented	Regenerated the committed reference baseline, which still described a 9-scenario frozen set after three sre.* scenarios made it 12. Every published metric, sample size, and confidence interval therefore described a set that no longer existed; routed_correctly_rate was 0.111 and is 0.083. The baseline test now derives the scenario count and the t2 economics from the set instead of pinning 9 and “exactly one t2 scenario”, so the next addition fails loudly at the artifact rather than silently in the numbers.	current change; docs/baselines/v2026.07.{json,md} and the Korean pair regenerated by tools.baseline_run; the core and shared-package suites passed 11913 cases with 131 skips, including the previously red test_baseline_runner and test_models_facade_only.	The baseline remains synthetic-harness evidence and is not claim eligible; a live baseline and treatment cohort is still the open item below.
2026-08-19	in-progress	Gave the sre pack its third coverage dimension with asserted evidence rather than a manifest entry. A dedicated test replays sre.cluster-diagnostics-missing.001 against a publisher that drops the first request, so the effect outcome is genuinely unknown, then proves the run closes a terminal publish_outcome_unknown audit entry before the error escapes, records no PR, caches nothing, and that a retry over the same executor publishes exactly one observation mode PR. This is the first partial_failure_recovery evidence in any pack.	current change; tests/scenarios and test_shadow_eval.py passed 116 focused cases; the new test is mutation-verified - making _close_unknown_publish a no-op fails it with assert 0 == 1.	successful_full_loop, cross_objective_conflict, and a3e_or_non_applicability remain unevidenced for sre. A full-loop claim needs independent effect verification, which observation mode execution does not provide, and an A3-E claim needs the unwired standing-authority evaluator. The manifest check still only proves a cited test exists, not that it asserts its dimension.
2026-08-14	in-progress	Adopted the implementation ledger without reconstructing earlier provenance and separated executable measurement mechanics from unproven outcome claims.	current change; measurement source, focused tests, scenario manifest, and constitutional register cited above.	Complete scenario coverage and retain a live baseline/treatment cohort with authoritative outcome closure.
2026-08-18	in-progress	Gave the sre capability pack its first scenarios, so no pack is missing. Three frozen scenarios replay through the real control loop with the shipped catalog: an observability precondition that fires kubernetes-cluster.diagnostic-settings-required and opens a observation mode PR, an error-budget burn on an unmodelled target that holds for review and publishes nothing, and telemetry retention above the reviewed ceiling. One scenario per existing domain keeps the balance check satisfied. Only unknown_or_deny and deterministic_replay_with_evidence are claimed, because the remaining four dimensions have no evidence yet; the pack stays partial and the set stays incomplete.	current change; tests/scenarios passed 98 focused cases, including all three new replays through ControlLoop.process against the shipped rules, policies, and ActionTypes.	Author the successful-full-loop, cross-objective-conflict, partial-failure-recovery, and A3-E-or-non-applicability cases for every pack. A full-loop claim additionally needs independent effect verification, which observation mode execution does not provide.

Remaining work

• Complete all five constitutional scenario packs with success, deny or unknown, conflict, partial-failure recovery, A3-E or non-applicability, and deterministic replay cases.
• Retain one reference baseline and FDAI treatment on the identical frozen scenario set with sample sizes, confidence intervals, absolute values, and no unsupported multiplier claim.
• Bind live incident, change, cost, human-touchpoint, and independently verified outcome records into the KPI projection, then prove all zero-threshold guards remain zero.

Primary Objective

Minimize human intervention in cloud operations across three initial verticals under an AIOps approach - Resilience, Change Safety, and Cost Governance - by resolving most events deterministically (T0/T1) and reserving LLM inference (T2) for the residual ambiguous minority, without regressing the guard metrics. Autonomy that improves a success metric while degrading a guard metric is a failure, not a win.

SRE is the operating model across the three verticals. Disaster recovery and Chaos Engineering are Resilience capabilities, Architecture Review Board governance applies across domains, and FinOps is the Cost Governance discipline.

Accuracy contract

FDAI does not claim that every novel diagnosis is correct. It targets 100% contract-conformant behavior: an agent either produces a schema-valid, evidence-supported, authorized result or records an explicit unknown, no-op, denial, rollback, or human-review outcome. The platform target is zero unsafe guesses, not forced answers.

The following violations have a release threshold of exactly zero:

• action against the wrong object identity or stale target revision;
• execution outside the registered ActionType, standing authority, or impact scope;
• success claimed from a broker/API receipt without independent effect verification;
• external state asserted from an ontology write rather than an authoritative observation;
• learning output that raises authority without review and promotion evidence.

Autonomy before human review

An unresolved event does not immediately become a human task. Within its bounded deadline, FDAI tries fresh evidence acquisition, an alternate authoritative source, deterministic reevaluation, verified pattern reuse, a smaller safe plan, no-op, or pre-authorized recovery. Human review begins only when ambiguity remains, policy mandates approval, or risk exceeds standing authority. Every attempt shares the event correlation and contributes no additional human touchpoint.

Definitions

Terms used across all metrics, fixed here to avoid ambiguity:

• Event: one normalized, deduplicated item entering the control loop (post event-ingest), identified by its stable idempotency key. All per-event rates are computed over this unit.
• Scenario set: a frozen, versioned collection spanning SRE, ARB / Change Safety, FinOps / Cost Governance, DR, and Chaos Engineering capability packs, used identically for baseline and treatment. Each release records the scenario-set and per-pack versions (e.g. v2026.07).

Current coverage gap: services/core-control-plane/tests/scenarios/manifests/v2026.07.json assigns every fixture to SRE, ARB / Change Safety, FinOps, DR, or Chaos. A coverage dimension counts only when it cites a scenario owned by that pack and an existing executable test. The set remains incomplete: SRE has no scenario and every existing pack lacks one or more required cases. FDAI must not claim complete domain coverage until all five packs are complete.

• Reference agent: the fixed comparison system (documented, single-model, no tiering) measured in Phase 0. Its version is pinned per baseline run.
• Human touchpoint: any action requiring a human decision or input (human approval approval, manual edit, manual rollback). Each uniquely identified action or approval counts once, while repeated lifecycle rows for the same action or approval don’t add another touchpoint. One event can contribute more than one touchpoint. Read-only viewing of the console is not a touchpoint.
• Auto-resolved event: an event that reaches a terminal, correct outcome with zero human touchpoints and no post-hoc rollback within the measurement window. An executor dispatch is pending, not resolved, until an explicit measurement.action_outcome.v1 record closes the observation with enforcement mode, passed verification, an auto decision, and no rollback.
• Measurement window: the fixed observation period per run (default: 30 days rolling, or one full scenario-set replay), stated with every reported figure.
• Contract-conformant outcome: one terminal result whose target, evidence, authority, action, effect verification, and audit records satisfy their exact versioned contracts. An explicit unknown or safe no-op is conformant; an unsupported success is not.

Success Metrics

Each metric fixes a unit, formula, and reporting window. Targets are relative to the reference agent on the same scenario-set version and are directional targets pending measurement.

#	Metric	Precise definition	Unit	Direction	Target vs baseline
1	Cost per unit	total attributable spend ÷ units processed, computed separately as $/incident, $/change, $/optimization	USD/unit	lower is better	large reduction (state factor only when measured)
2	Auto-resolution rate	auto-resolved events ÷ total events, in [0, 1]	ratio	higher is better	5× the baseline ratio (capped at 1.0)
3a	MTTR	mean(resolve_time − detect_time) over resolved incidents	seconds	lower is better	5× shorter (0.2× baseline)
3b	Change lead time	mean(merge_time − change_request_time) over changes	seconds	lower is better	5× shorter (0.2× baseline)
4	Human intervention	human touchpoints ÷ (total events ÷ 100)	touchpoints / 100 events	lower is better	0.2× baseline (i.e. 1/5)

Notes:

• Metric 1 cost includes model inference, compute, storage, and event-bus spend attributable to processing; it excludes fixed platform overhead shared with non-FDAI workloads.
• MTTR and lead time are reported as median and p90 alongside the mean, because latency distributions are skewed and a mean alone hides tail regressions.
• A 5× target on a ratio (metric 2) is bounded: report both the multiplier and the absolute ratios, since a multiplier is meaningless once the baseline is already high.

Guard Metrics (must not regress)

Guard metrics veto a promotion: any breach demotes the action from enforcement mode back to observation mode. Each has an explicit threshold, not just a direction.

Guard metric	Definition	Threshold
Change failure rate (CFR)	changes causing incident/rollback ÷ total changes	≤ baseline CFR (no increase)
False-positive rate	incorrect actions ÷ actions taken	≤ baseline; alert if > baseline + 1pp
False-negative rate	missed true events ÷ true events	≤ baseline; alert if > baseline + 1pp
Rollback rate	actions rolled back ÷ actions executed	≤ baseline rollback rate
Policy-violation escapes	autonomous actions that violate policy and reach enforce	exactly 0 (any escape blocks release)
Wrong-target or stale-revision execution	actions applied to a different object or revision than the approved plan	exactly 0
Unauthorized execution	actions outside registered type, identity, standing authority, or impact scope	exactly 0
Unverified success claims	actions reported successful without independent expected-effect closure	exactly 0

Thresholds are evaluated on the same measurement window and scenario-set version as the success metrics, so a gain and a guard breach are never compared across different data.

Leading vs Lagging Indicators

Success metrics 1-4 are lagging (observable only after enough events resolve). Promotion decisions also watch leading indicators that predict guard-metric health earlier:

• per-tier coverage share (T0 70-80%, T1 15-20%, T2 5-10%) drifting out of band,
• mixed-model disagreement rate (T2 quality gate) trending up,
• verifier hold for review/fail rate rising,
• observation mode-vs-enforce decision divergence for a candidate action.

Leading indicators trigger investigation before a lagging guard metric regresses.

Measurement-First Rule

• No autonomy ships without telemetry to measure its effect (metrics 1-4 and all guard metrics).
• Phase 0 establishes the KPI dashboard and the reference baseline before any tier goes live (phase-0-instrumentation.md).
• Multiplier claims (2-4) are only stated after the baseline and the treatment are both measured under the identical, frozen scenario-set version.
• Statistical validity: report each factor with a sample size (event count), a confidence interval, and the scenario-set version. Differences within the confidence interval are reported as “no measured change”, not as an improvement. A zero-sample Wilson interval is [0, 1] (unknown), never evidence that accuracy is exactly zero.
• Operational promotion evidence: bind frozen benchmark and live-observation mode samples to one full FDAI revision, ActionType digest, scenario case, and authoritative measurement unit. Latest corrections replace prior rows without changing cohort, scenario, observation time, or causal lineage. Separate frozen/live Wilson 95% lower bounds, distinct live days, zero escapes, executed-action rollback and complete recurrence windows, verified causal receipts, and Dynamic review must pass. A closed causal receipt counts only with confirmed closure. Raw metrics cannot promote; a verified receipt permits a separate review only.
• Fairness: baseline and treatment run the same scenarios, the same input distribution, and the same measurement window; the reference agent is not deliberately handicapped.

Data Collection and Telemetry

Every metric maps to a concrete telemetry source so the dashboard is buildable, not aspirational:

• Structured events + traces (OpenTelemetry) carry event_id, tier, decision, mode (observation mode/enforce), and timestamps - sourcing metrics 2, 3a/3b, and leading indicators.
• Append-only audit log sources human touchpoints (metric 4), rollbacks, and policy escapes.
• Outcome finalization records (measurement.action_outcome.v1) are the authority for auto-resolution. Dispatch-only events remain pending, verified non-rollback outcomes enter the finalized denominator, and rollback/adverse outcomes remain visible without becoming successes. When an action has corrected finalization rows, only its highest audit sequence is authoritative; an explicit verification failure remains a rejected observation rather than disappearing.
• Explicit metric observations use the latest row for each event_id and metric key. A retry or correction for one event replaces that event’s earlier value instead of adding statistical weight; observations from different events remain independent samples.
• MTTR (metric 3a) is computed by the pure aggregator core/measurement/mttr.py, which folds resolved incidents (resolved_at - opened_at) into mean, median, and p90 seconds; unresolved and integrity-violating incidents are counted but excluded, never contributing a 0 or a negative duration. The delivery-layer wiring that feeds it live incidents (replacing the synthetic dev value in the /kpi/autonomy panel) is tracked as follow-up.
• Cost/usage records (model tokens, compute time, storage, bus throughput) source metric 1; attribution keys spend to the originating event_id. Repeated lifecycle rows for one action contribute the latest observed savings value once rather than weighting or summing retries.
• All metric inputs are English, secret-free, and customer-agnostic per the repo scope rules.

Review Cadence

• Per promotion: no action moves observation mode → enforce without a passing metrics + guard review.
• Weekly: dashboard review of leading indicators and guard-metric drift.
• Per scenario-set version bump: full baseline re-measurement so targets track a current, fair reference rather than a stale one.

Where the Target Multipliers Would Come From

The mechanisms below are the hypothesized sources of the targeted gains; each is only credited once measured against the baseline. Framing is intentionally “uses the LLM less”, not “a smarter LLM”.

Target	Hypothesized mechanism
Auto-resolution ↑	T0/T1 deterministically close the ~85-90% majority of events; fewer escalations to T2/human approval.
MTTR / lead time ↓	T0/T1 have no LLM round-trip (ms-s); auto-fix PRs remove human wait time.
Human intervention ↓	safety check auto-approves low-risk actions; learned T1 actions avoid repeat human touch.
Cost per unit ↓	only ~5-10% of events reach a frontier model; OSS/CSP-neutral stack; event-driven scale-to-zero.

Core insight: the gains are hypothesized to come from a structure that uses the LLM less, not from a smarter LLM - and this claim stands or falls on the Phase 0 measurement.

Next steps

To learn about	Read
How the baseline is instrumented	phases/phase-0-instrumentation.md
Per-tier coverage targets and the trust router	../../.github/instructions/architecture.instructions.md
Safety invariants that guard-metrics enforce	../../.github/instructions/coding-conventions.instructions.md
The KPI dashboard shipped with P0	../dashboards/phase-0-kpi.json