Goals and Metrics

Use these goals and metrics to evaluate autonomy with proof. Every autonomy claim is backed by a measured baseline; nothing is asserted from estimation. Improvement factors below (5×, large reduction, 1/5) are targets, not achieved results - they may only be stated as achieved once both the reference baseline and the FDAI treatment have been measured on the same scenario set (see the measurement-first rule).

This document is the source of truth for KPIs. It aligns with the tier coverage targets in architecture.instructions.md and is operationalized by phase-0-instrumentation.md.

Primary Objective

Minimize human intervention in cloud operations across three initial verticals under an AIOps approach - Resilience, Change Safety, and Cost Governance - by resolving most events deterministically (T0/T1) and reserving LLM inference (T2) for the residual ambiguous minority, without regressing the guard metrics. Autonomy that improves a success metric while degrading a guard metric is a failure, not a win.

SRE is the operating model across the three verticals. Disaster recovery and Chaos Engineering are Resilience capabilities, Architecture Review Board governance applies across domains, and FinOps is the Cost Governance discipline.

Accuracy contract

FDAI does not claim that every novel diagnosis is correct. It targets 100% contract-conformant behavior: an agent either produces a schema-valid, evidence-supported, authorized result or records an explicit unknown, no-op, denial, rollback, or human-review outcome. The platform target is zero unsafe guesses, not forced answers.

The following violations have a release threshold of exactly zero:

action against the wrong object identity or stale target revision;
execution outside the registered ActionType, standing authority, or impact scope;
success claimed from a broker/API receipt without independent effect verification;
external state asserted from an ontology write rather than an authoritative observation;
learning output that raises authority without review and promotion evidence.

Coverage claim contract

FDAI reports operational coverage with one immutable OperationalCoverageReceipt per governed universe. Supported domains are asset inventory, governance evaluation, operating scope, incident diagnosis, fix effects, and knowledge evidence check.

Each receipt pins the scope, denominator, and evidence by digest. Every denominator item receives one terminal coverage disposition: covered, unknown, stale, unsupported, inaccessible, conflicting, or invalid. A governance result can be compliant or noncompliant and still be covered; policy outcome is separate from whether FDAI evaluated the item with current evidence.

The receipt computes coverage in integer basis points. It can meet a target only when all denominator items are accounted for, the evidence is fresh at evaluation time, the covered count meets the configured threshold, and every configured zero-tolerance disposition has count zero. The receipt is measurement evidence only and grants no approval, mutation, or execution authority.

Autonomy before human review

An unresolved event does not immediately become a human task. Within its bounded deadline, FDAI tries fresh evidence acquisition, an alternate authoritative source, deterministic reevaluation, verified pattern reuse, a smaller safe plan, no-op, or pre-authorized recovery. Human review begins only when ambiguity remains, policy mandates approval, or risk exceeds standing authority. Every attempt shares the event correlation and contributes no additional human touchpoint.

Definitions

Terms used across all metrics, fixed here to avoid ambiguity:

Event: one normalized, deduplicated item entering the control loop (post event-ingest), identified by its stable idempotency key. All per-event rates are computed over this unit.
Benchmark scenario set: a frozen, versioned synthetic collection spanning SRE, ARB / Change Safety, FinOps / Cost Governance, DR, and Chaos Engineering capability packs. It tests deterministic behavior and release regressions. It does not become live evidence merely because a deployed runtime replays it.
Operational cohort protocol: a versioned measurement contract frozen before observation. It fixes eligibility, source authority, arm assignment, window length, revision, metric denominators, independence keys, interval methods, and a bounded stopping rule. Baseline and treatment can contain different real events, but both use the same protocol and neither event is executed through both arms.

Current coverage gap: services/core-control-plane/tests/scenarios/manifests/v2026.10.json records complete frozen coverage for SRE, ARB / Change Safety, FinOps, DR, and Chaos. Each pack cites a reviewed capability-specific outcome test in addition to inherited deterministic replay evidence. FDAI must not claim complete domain coverage for a deployed runtime or measured production baseline until those separate operational receipts exist.

Reference agent: the fixed comparison system (documented, single-model, no tiering) measured in Phase 0. Its version is pinned per baseline run.
Human touchpoint: any action requiring a human decision or input (human approval approval, manual edit, manual rollback). Each uniquely identified action or approval counts once, while repeated lifecycle rows for the same action or approval don’t add another touchpoint. One event can contribute more than one touchpoint. Read-only viewing of the console is not a touchpoint.
Auto-resolved event: an event that reaches a terminal, correct outcome with zero human touchpoints and no post-hoc rollback within the measurement window. An executor dispatch is pending, not resolved, until an explicit measurement.action_outcome.v1 record closes the observation with enforcement mode, passed verification, an auto decision, and no rollback.
Measurement window: the fixed observation period per run (default: 30 days rolling, or one full scenario-set replay), stated with every reported figure.
Contract-conformant outcome: one terminal result whose target, evidence, authority, action, effect verification, and audit records satisfy their exact versioned contracts. An explicit unknown or safe no-op is conformant; an unsupported success is not.

Success Metrics

Each metric fixes a unit, formula, and reporting window. Targets are relative to the reference agent on the same scenario-set version and are directional targets pending measurement.

#	Metric	Precise definition	Unit	Direction	Target vs baseline
1	Cost per unit	total attributable spend ÷ units processed, computed separately as `$/incident`, `$/change`, `$/optimization`	USD/unit	lower is better	large reduction (state factor only when measured)
2	Auto-resolution rate	auto-resolved events ÷ total events, in `[0, 1]`	ratio	higher is better	5× the baseline ratio (capped at 1.0)
3a	MTTR	mean(resolve_time − detect_time) over resolved incidents	seconds	lower is better	5× shorter (0.2× baseline)
3b	Change lead time	mean(merge_time − change_request_time) over changes	seconds	lower is better	5× shorter (0.2× baseline)
4	Human intervention	human touchpoints ÷ (total events ÷ 100)	touchpoints / 100 events	lower is better	0.2× baseline (i.e. 1/5)

Notes:

Metric 1 cost includes model inference, compute, storage, and event-bus spend attributable to processing; it excludes fixed platform overhead shared with non-FDAI workloads.
MTTR and lead time are reported as median and p90 alongside the mean, because latency distributions are skewed and a mean alone hides tail regressions.
A 5× target on a ratio (metric 2) is bounded: report both the multiplier and the absolute ratios, since a multiplier is meaningless once the baseline is already high.

Guard Metrics (must not regress)

Guard metrics veto a promotion: any breach demotes the action from enforcement mode back to observation mode. Each has an explicit threshold, not just a direction.

Guard metric	Definition	Threshold
Change failure rate (CFR)	changes causing incident/rollback ÷ total changes	≤ baseline CFR (no increase)
False-positive rate	incorrect actions ÷ actions taken	≤ baseline; alert if > baseline + 1pp
False-negative rate	missed true events ÷ true events	≤ baseline; alert if > baseline + 1pp
Rollback rate	actions rolled back ÷ actions executed	≤ baseline rollback rate
Policy-violation escapes	autonomous actions that violate policy and reach enforce	exactly 0 (any escape blocks release)
Wrong-target or stale-revision execution	actions applied to a different object or revision than the approved plan	exactly 0
Unauthorized execution	actions outside registered type, identity, standing authority, or impact scope	exactly 0
Unverified success claims	actions reported successful without independent expected-effect closure	exactly 0

Thresholds are evaluated on the same measurement window and scenario-set version as the success metrics, so a gain and a guard breach are never compared across different data.

Leading vs Lagging Indicators

Success metrics 1-4 are lagging (observable only after enough events resolve). Promotion decisions also watch leading indicators that predict guard-metric health earlier:

per-tier coverage share (T0 70-80%, T1 15-20%, T2 5-10%) drifting out of band,
mixed-model disagreement rate (T2 quality gate) trending up,
verifier hold for review/fail rate rising,
observation mode-vs-enforce decision divergence for a candidate action.

Leading indicators trigger investigation before a lagging guard metric regresses.

Measurement-First Rule

No autonomy ships without telemetry to measure its effect (metrics 1-4 and all guard metrics).
Phase 0 establishes the KPI dashboard and the reference baseline before any tier goes live (phase-0-instrumentation.md).
Multiplier claims (2-4) are only stated after the baseline and treatment are measured from real operations under one frozen operational cohort protocol. Synthetic benchmark replay can block a release, but it cannot make an operational claim eligible.
Statistical validity: report each factor with a sample size (event count), a confidence interval, and the scenario-set version. Differences within the confidence interval are reported as “no measured change”, not as an improvement. A zero-sample Wilson interval is [0, 1] (unknown), never evidence that accuracy is exactly zero.
Operational promotion evidence: bind frozen benchmark and live-observation mode samples to one full FDAI revision, ActionType digest, scenario case, and authoritative measurement unit. Latest corrections replace prior rows without changing cohort, scenario, observation time, or causal lineage. Separate frozen/live Wilson 95% lower bounds, distinct live days, zero escapes, executed-action rollback and complete recurrence windows, verified causal receipts, and Dynamic review must pass. A closed causal receipt counts only with confirmed closure. Raw metrics cannot promote; a verified receipt permits a separate review only.
Fairness: baseline and treatment use the same precommitted eligibility taxonomy, source mapping, window length, metric definitions, independence rule, and stopping rule. The baseline is the actual observed non-FDAI operating process, not the always-human-review test stub. Treatment is the deployed FDAI revision. A historical-baseline/prospective-treatment design avoids executing two competing processes against one live event.
Source binding registry: the trusted policy maps every required metric and guard in each arm to exactly one in-repository exporter workflow and stable source id. The protected importer injects that source id into observation provenance; the artifact cannot declare it. One workflow can own several measures, and separate workflows can represent different products without changing Core. Once an arm has any binding, it must cover every required measure. Partial coverage, duplicate ownership or source ids, reuse of one workflow across both arms, missing workflow files, duplicate policy keys, or artifact-declared source authority is blocked.
Metric denominators: event rates, resolved incidents, merged changes, and attributable cost units have different natural denominators. Each required metric and guard retains its own effective sample size. An arm’s published floor is the smallest effective sample size among all required measures, so a large event count cannot hide an undersized MTTR, change, or cost sample.
Intervals and independence: Wilson 95% intervals apply to Bernoulli rates. Deterministic bootstrap 95% intervals apply to skewed continuous and count means. Repeated records and events sharing one declared independence key contribute once to that measure.
Stopping rule: each arm has a bounded 90-day observation window. If any required effective sample size remains below 30 at the cutoff, the producer records an undersized result and the claim stays ineligible.

Data Collection and Telemetry

Every metric maps to a concrete telemetry source so the dashboard is buildable, not aspirational:

Dashboard event accounting uses one versioned measurement.control_loop.v1 record for a nonduplicate normalized event’s final control-loop classification. Intermediate evaluations, source transitions, and generic audit rows are not additional events. This record preserves the event identity, tier, route, mode, and time; it does not assert successful fix.
Live metric observations use versioned, source-bound records with explicit units, observation identity, event attribution, and effective time. The Operator reads the durable facts through its own role and never imports Core implementation code. Missing, contradictory, truncated, or unverified evidence cannot be converted into a measured zero. Descriptive live values remain separate from admitted baseline/treatment comparisons and promotion eligibility.
Structured events + traces (OpenTelemetry) carry event_id, tier, decision, mode (observation mode/enforce), and timestamps - sourcing metrics 2, 3a/3b, and leading indicators.
Append-only audit log sources human touchpoints (metric 4), rollbacks, and policy escapes.
Outcome finalization records (measurement.action_outcome.v1) are the authority for auto-resolution. Dispatch-only events remain pending, verified non-rollback outcomes enter the finalized denominator, and rollback/adverse outcomes remain visible without becoming successes. When an action has corrected finalization rows, only its highest audit sequence is authoritative; an explicit verification failure remains a rejected observation rather than disappearing.
Explicit metric observations keep the latest row per natural measurement unit and metric. Timing uses the source incident or change identity, so several events for one incident do not inflate its sample size. Cost remains attributed per normalized event. Retries and corrections replace prior values instead of adding statistical weight.
MTTR (metric 3a) is computed by the pure aggregator core/measurement/mttr.py, which folds resolved incidents (resolved_at - opened_at) into mean, median, and p90 seconds; unresolved and integrity-violating incidents are counted but excluded, never contributing a 0 or a negative duration. The protected measurement-source importer feeds independently admitted lifecycle facts to this reducer; unverified native lifecycle changes remain unavailable.
Cost/usage records (model tokens, compute time, storage, bus throughput) source metric 1; attribution keys spend to the originating event_id. Repeated lifecycle rows for one action contribute the latest observed savings value once rather than weighting or summing retries.
All metric inputs are English, secret-free, and customer-agnostic per the repo scope rules.

Review Cadence

Per promotion: no action moves observation mode → enforce without a passing metrics + guard review.
Weekly: dashboard review of leading indicators and guard-metric drift.
Per scenario-set version bump: full baseline re-measurement so targets track a current, fair reference rather than a stale one.

Where the Target Multipliers Would Come From

The mechanisms below are the hypothesized sources of the targeted gains; each is only credited once measured against the baseline. Framing is intentionally “uses the LLM less”, not “a smarter LLM”.

Target	Hypothesized mechanism
Auto-resolution ↑	T0/T1 deterministically close the ~85-90% majority of events; fewer escalations to T2/human approval.
MTTR / lead time ↓	T0/T1 have no LLM round-trip (ms-s); auto-fix PRs remove human wait time.
Human intervention ↓	safety check auto-approves low-risk actions; learned T1 actions avoid repeat human touch.
Cost per unit ↓	only ~5-10% of events reach a frontier model; OSS/CSP-neutral stack; event-driven scale-to-zero.

Core insight: the gains are hypothesized to come from a structure that uses the LLM less, not from a smarter LLM - and this claim stands or falls on the Phase 0 measurement.

Next steps

To learn about	Read
How the baseline is instrumented	phases/phase-0-instrumentation.md
Per-tier coverage targets and the trust router	../../.github/instructions/architecture.instructions.md
Safety invariants that guard-metrics enforce	../../.github/instructions/coding-conventions.instructions.md
The KPI dashboard shipped with P0	../dashboards/phase-0-kpi.json

To learn about	Read
Delivery status and remaining work	Implementation ledger