Observability and Detection

Learn how FDAI turns raw telemetry into detected issues the control loop can act on: event correlation, anomaly detection, predictive / forecasting, and root-cause analysis (RCA). These are the detection signals an AIOps platform is expected to provide - added here without breaking deterministic-first: every signal emits a normalized detected issue that flows through the existing trust-router → tiers → risk-gate → executor → audit path, never a side channel, and nothing auto-executes outside the safety check and the seven safeguards.

Reference: control loop, tiers, and the quality gate in architecture.instructions.md; measurement and guard metrics in goals-and-metrics.md; rule/signal sources in rule-catalog-collection.md; module placement and DI seams in project-structure.md; the prompt-injection threat model in security-and-identity.md. Correlation and out-of-band detection are introduced in phase-1-rule-catalog-t0.md; FinOps cost-anomaly and DR RPO/RTO forecasting land in phase-3-integrated-loop.md. Customer-agnostic; all examples are synthetic.

Design Stance (deterministic-first, not ML-first)

Detection is explainable and evidence-backed first: statistical baselines, thresholds, and correlation rules resolve most signals with no model call. Models (T1 similarity, T2 reasoning) enter only for fuzzy correlation and novel RCA - the same 5-10% budget.
A detection signal is a detected issue, not an action. It is routed and risk-gated like any event; a prediction or anomaly never auto-remediates on its own - it raises a observation mode detected issue or a fix PR that the safety check and human approval govern.
The canonical object type and one detected record are named Finding; a collection, page, or navigation label is named Findings. A Detected issue remains independent from an Incident unless the incident correlation and lifecycle gates promote it. Agent Activity can show that an agent created or updated a Detected issue, with time and correlation evidence, but it does not own Detected issue status, deduplication, severity, or Incident promotion. A cross-domain Findings projection owns those operator concerns; until that projection is implemented, the owning domain projection and audit record remain authoritative rather than the activity timeline.
Routine monitoring is not an Incident. A healthy heartbeat, successful probe, or within-threshold sample records observation evidence only. An Incident can open only after a detector emits a bounded, grounded detected issue and IncidentLifecycleWorkflow rechecks the allowed agent principal, correlation keys, reason, and member-event evidence.
A repeated-event burst is an anomaly, not automatic Incident authority. Candidate handoff requires incident_correlation=correlate, non-empty correlation and evidence keys, and the configured minimum severity; none never opens an Incident. Huginn stamps trusted UTC ingestion and accepts only source time no later than that boundary, so replay or future time cannot manipulate Heimdall’s bounded episode. Distinct evidence keys count once, burst severity is monotonic, and one accepted episode suppresses duplicate candidates while allowing a more-severe update. A quiet interval creates a new opaque episode; restart reuses only an active Incident with the exact non-episode correlation keys. Failed publication or handoff retries only on a matching Event, with accepted and held kept distinct. Candidate text, evidence count, and lifecycle notices remain bounded and append-only.
Analyzer detected issues are bounded detector outputs. Deployed and local loops use the same fixed-rate cadence and publish one retry-stable Event per logical Resource, signal, and one-minute occurred_at bucket; the five-minute analysis window remains independent. UUID5 keys, distinct evidence, and one anomaly key per episode and severity prevent polling, replay, and handoff retries from multiplying judgment. Future or timezone-naive detected issues, malformed evidence, partial target coverage, and publication or receipt failures fail before a clean result. Inventory-backed analysis preserves ontology Resource.id and its verified canonical Resource type in the detected issue Event so the trust router can select the lowest sufficient tier. An exact active-snapshot provider reference scopes only the metric query and never enters Detected issue, receipt, Incident, or serialized error evidence. Configured and legacy provider identities reconcile within one inventory generation or fail closed, and inventory-free metric targets require both identities. Prometheus is eligible only when composed queries and responses preserve the exact requested identity; otherwise Azure Monitor Logs remains the path.
Complete inventory promotion also triggers configuration-rule evaluation. After the ontology projection reports a complete available generation, the Inventory Job publishes one safe-to-retry inventory.resource_observed Event per Resource. SignalType dispatch resolves that Event to resource.configuration.observed, so T0 evaluates the applicable active Rego rules against the observed properties. The generation and Resource content determine the Event identity. A retry of the same generation is deduplicated, while a later generation is evaluated again. An incomplete projection or publication failure remains pending and does not report a clean observation.
Complete inventory evaluation needs an explicit terminal-outcome contract before the Console can display an evaluated Rule detected issues summary. Inventory remains the observation producer. Forseti records one versioned terminal outcome for every eligible (generation, resource, rule) tuple against the exact active catalog and Rule revisions. The outcome is compliant, violated, or abstained; abstained is used when the evidence is missing, stale, unsupported, or otherwise cannot support a deterministic rule result. A separate completion record closes the generation only when the expected denominator is covered and every outcome is bound to an evaluation receipt and Saga audit reference. An empty complete inventory can close only with an expected denominator of zero. Provider, broker, OPA, persistence, or projection failure leaves evaluation unavailable and preserves the prior complete summary.
Heimdall bounds retained repeated-event episodes globally and per resource. A correlation flood from one resource evicts only that resource’s oldest episode before it can displace another resource’s partially accumulated evidence.
New detectors ship in observation mode and are promoted per the observation mode→enforce rule; their accuracy and false-positive rate are measured against the Phase 0 baseline.

Baseline evaluation terminal contract design

This section records the reviewed contract boundary for issue #1199 before implementation. The runtime behavior lands in ordered slices so each PR can be reviewed without claiming evidence the repository does not yet produce.

Initial design. Add two shared service-contract records:

BaselineEvaluationOutcome is the per-rule terminal record. It binds one inventory generation and digest, one resource identity and digest, one catalog revision, one Rule identity and revision, one terminal outcome (compliant, violated, or abstained), the evaluation receipt, the Saga audit reference, and a canonical outcome digest.
BaselineEvaluationCompletion is the per-generation close record. It binds the same generation and catalog revision to the expected denominator, observed outcome counts, completion receipt, Saga audit reference, and canonical completion digest.

Critique. A count-only summary is not enough because it can still infer zero from an incomplete denominator. A violation-only Finding stream is also insufficient because it omits compliant and held-for-review cases. The contract therefore needs per-rule terminal records, sorted identities, bounded reasons, and exact digest validation so duplicate or reordered delivery can be replayed without changing the covered denominator. The completion record must reject mismatched counts, mixed generation/catalog identities, and any authority flag. It also must allow the legitimate zero-resource case without allowing a producer to hide missing outcomes behind evaluated=true.

Revision. The shared contract is authority-free and additive. It does not change the fixed Pantheon roles, topic ownership, or executor path: inventory produces observations, Forseti judges the rule outcomes, Saga persists append-only audit, and the Operator projection reads only complete authoritative outcomes. A projection can replace the current unavailable summary only after a completion record proves the expected denominator for the active generation and catalog revision. If a later catalog revision appears, the prior complete summary remains visible as prior evidence until the new revision has its own complete denominator; it is never treated as current evaluation.

The first runtime slice keeps that separation. Forseti exposes a typed method for one complete inventory generation. It evaluates each eligible Resource with the active T0 engine and writes one terminal outcome per cited Rule only after an injected Saga audit binder returns the audit reference the shared contract requires. Missing, stale, or unsupported evidence becomes abstained. An empty complete generation writes a zero-denominator completion. The method writes deterministic StateStore records and changes no AgentSpec, topic owner, subscription, approval, execution, or projection authority.

The Saga slice supplies that binder. Saga appends a baseline-evaluation audit entry to its append-only chain and to the durable audit store when one is bound, then returns the stable audit reference and digest that Forseti records in each outcome or completion. The audit payload contains the inventory generation, resource identity, catalog revision, Rule revision, evaluation receipt, and terminal outcome supplied by Forseti. Saga still does not judge or project Rule detected issues; it only binds replayable audit evidence.

The Operator projection slice admits a Rule detected issues summary only from a complete BaselineEvaluationCompletion plus every matching BaselineEvaluationOutcome. It counts only violated outcomes as current detected issues, keeps compliant and held for review Rule refs at zero so resolved detected issues disappear from the current count, and rejects truncated, mismatched, or partial coverage instead of inferring zero. A previously stored complete summary remains readable while a newer generation is incomplete or unavailable; absent authoritative records still render evaluated: false.

Known limitation. The version 1 completion sets its expected denominator to the number of outcomes Forseti wrote, so it can’t detect an omitted Resource and Rule pair. It also identifies an inventory observation rather than a Rule activation generation. A version 2 completion that derives expected pairs independently, and the migration of the Rule detected issues summary to it, are designed in Framework Rule Evidence from T0. No runtime path calls the Forseti evaluation yet, and the version 1 audit binding calls the Saga agent directly for each record; the baseline trigger design replaces both. Until then, treat the version 1 summary as observed counts, not proof of complete coverage.

Frozen configuration baseline checks

Configuration drift is a T0 (deterministic rules) detected issue. A reviewed actual snapshot is frozen as intended state and is never replaced automatically by a later observation. The human-readable DOCX and canonical JSON baseline carry the same version, scope, creation time, and document digest. Generation and validation also verify that every displayed resource, attribute, evidence gap, topology link, exception, and unknown item in canonical JSON appears in the paired DOCX. A matching file digest alone does not establish cross-format equivalence.

core/detection/configuration_drift.py canonicalizes resources, topology links, and comparable attributes, then reports added, removed, changed, unchanged, unknown, or unauthorized.
A partial snapshot cannot prove removal. Missing resources, links, or attributes remain blocked until an authoritative source supplies complete evidence.
The configured baseline version, SHA-256 digest, and scope are server-owned. A caller cannot select another target through tool arguments.
An immutable baseline registry can hold candidate, active, superseded, and archived versions for multiple scopes. It allows one active version per scope. Active and replay-pinned sources are selected by server composition, never by conversational input, and the registry exposes no mutation API.
delivery/configuration_drift.py retains JsonFileConfigurationObservationSource for development and evidence replay. delivery/azure/configuration_drift.py adds a read-only Azure Resource Graph source for one server-owned scope. Deployment configuration selects at most 64 scalar attribute paths through a strict identifier grammar. The adapter constructs the query, returns only selected values, marks missing values as unknown, uses non-empty presence tokens so an omitted false value cannot alias an explicitly present empty scalar, replaces provider ids with stable digest suffixes, and rejects partial, oversized, malformed, or out-of-scope results. It does not infer topology or collect arbitrary provider property bags. Runtime bootstrap binds the source only when FDAI_CONFIGURATION_DRIFT_ENABLED is explicit and every scope, baseline, subscription, and attribute prerequisite is valid. A protected Core service transition reconstructs the reviewed snapshot from a bounded protected secret envelope on the VNet-integrated deployment runner, checks the exact canonical digest and expected resource count, and stores the content only as an immutable, content-addressed private Blob. The repository and service plan contain only the binding metadata. Core reads that Blob through its Managed Identity and verifies provider-valid digest metadata, and post-apply verification reads the deployed binding and Blob independently before comparing a fresh Azure Resource Graph observation. The retained receipt includes only digests, completeness, the decision, counts, and zero-valued authority counters. The Container Apps module emits no drift configuration by default. Once a deployment owner supplies both protected baseline envelope values, every later Core plan installs the pinned evidence toolchain, preserves that exact binding, and every successful Core apply repeats independent readback.
Knowledge retrieval explains and cites the reviewed document. It does not decide the drift. If Knowledge is unavailable, the deterministic report remains valid and the citation status stays blocked rather than being reported as supported. Each citation identity includes the exact baseline version and full DOCX SHA-256 digest so a reused filename cannot alias another document. Exact metadata lookup takes precedence; a bounded deterministic lexical fallback ranks chunks only within that pinned document and returns no result for an unrelated query. Provider exceptions emit a structured warning containing the exception type and pinned baseline identity, but never the exception message or chunk content.
The read-only capability reports mutation, approval, mitigation, and unsupported-claim counts. Each remains zero for a configuration check.
The public bind_configuration_drift composition helper installs this one server-pinned A0 capability through the immutable capability runtime. It does not add an ActionType, executor identity, schedule authority, or caller-selected scope.
Every fresh run records baseline-load, observation, comparison, Knowledge, and total latency plus resource and detected issue counts. Current observations are not reused through a TTL cache because a cached snapshot cannot satisfy a current-state question. The receipt rejects stage latencies that exceed total elapsed time beyond floating-point timer tolerance.
The configured Core runtime persists each completed check to the existing runtime:configuration-baseline: projection before returning its report. One revision-fenced, audited row per scope retains the latest observation, including failed or blocked decisions; older completions cannot replace newer evidence. Operator reads stored metadata and measured latency only, without invoking Azure on a Console GET. Missing baseline bindings still require deployment-owned configuration. A recorded check does not fabricate baseline version history, indexed Knowledge, a review campaign, or schedule approval. Console preserves blocked and unconfigured citation states instead of rendering a measured zero.
A pure review reducer accepts three idempotent run receipts for one pinned baseline. Only three verified runs can produce an inert weekly schedule proposal. Any blocked or unsafe run pauses the campaign, and the reducer never creates a scheduler task directly. A revisioned StateStore adapter persists campaign progress with atomic state-and-audit create and compare-and-set advance.
Before campaign advance, an immutable StateStore report ledger records the full detected issues, citations, safety counters, and measured performance under the campaign and run identity. Its strict codec supports restart replay, duplicate content is a no-op, and identity reuse with different evidence is blocked.
A ready campaign submits an inert Automation Blueprint. Independent review and the authenticated scheduler command remain mandatory before a observation mode weekly event exists. Configuration drift does not call the scheduler store or executor directly.

1. Event Correlation

A stage in event-ingest, immediately after normalize + deduplicate (see project-structure.md and phase-1-rule-catalog-t0.md): group related raw events into a single incident so downstream tiers reason about one thing, not a storm.

Deterministic-first: correlate by shared keys (resource id, deployment id, trace/correlation id, causal parent) within a bounded time window, using rules; fall back to T1 embedding similarity only for fuzzy grouping.
Grouping, not causation: correlation only asserts events belong together; a shared window can be coincidental. Assigning the cause is RCA’s job (section 4), never correlation’s.
Windowing and late arrival: the correlation window is configured per signal class; a late/out-of-order event matching an open incident’s keys is attached to it (or, past the window, opens a linked follow-on incident) - events are never silently dropped, and the per-resource ordering from architecture.instructions.md is preserved.
Idempotent grouping: the incident id is derived deterministically from the correlation keys, so reprocessing the same members yields the same incident regardless of arrival order.
Noise reduction: a burst of alerts from one root event collapses to one incident. This is reported as a measured noise-reduction ratio (incidents ÷ raw alerts), not an asserted gain, and no data is lost - members stay linked in the audit.
Output: one correlated incident event carrying its member event ids and a stable idempotency key; ordering/idempotency keys are preserved.
Lifecycle boundary: a correlation id is an investigation key, not proof that an Incident exists. IncidentRegistry owns the lifecycle record. Audit-only local fixtures remain available to Audit and Trace but are excluded from the operational Incident roster.
Operational-work policy: normalized Events declare incident_correlation. The default correlate preserves incident grouping. Discovery, inventory, scheduler, and workflow-control producers set none, retain correlation_id for trace and audit, and derive no Incident ID.
Upstream implementation: core/event_ingest/correlator.py (EventCorrelator) derives the incident anchor deterministically from an event’s correlation-id (or resource ref) plus a time-window bucket via incident_id_for; a burst sharing a key in one window collapses to one incident and a new window opens a linked follow-on. An event with incident_correlation=none or no anchor is reported correlated=False (never dropped). The keys feed IncidentRegistry.open, which accumulates membership idempotently.

2. Anomaly Detection

Generalizes the existing FinOps cost-anomaly hook (see phase-3-integrated-loop.md) to any metric stream - performance, reliability, security, and cost.

Method: statistical baselines (rolling and/or seasonal, with the seasonality window as config) and deviation thresholds (e.g. z-score or robust percentile bands), computed per signal class. Deterministic and explainable; the baseline, the deviation magnitude, and its direction (over/under) are recorded so a human can see why it fired.
Cold-start: a detector without enough baseline history to be reliable holds for review (stays in observation mode and emits no detected issue) rather than firing on a thin baseline; the cold-start suppression is counted as a metric, not hidden.
Categories: detected issues normalize to the canonical category enum (security | reliability | cost | config_drift | compliance) shared with the rule catalog - performance signals (latency/error-rate/saturation) and replication lag map to reliability, unusual access patterns to security, spend run-rate to cost. Severity derives from deviation magnitude.
Change-aware suppression: anomalies coincident with an in-flight change/maintenance window are correlated with the originating change event and suppressed or annotated, so a deploy does not manufacture false positives.
False-positive and false-negative control: debounce/settling windows plus a measured false-positive rate and false-negative (missed-anomaly) rate that a new detector must not regress - both map to guard metrics in goals-and-metrics.md.
Output: an anomaly detected issue that re-enters event-ingest (for an idempotency key and dedup) and then the trust router like any event.
Upstream implementation: core/detection/anomaly.py (MetricAnomalyDetector) ships the deterministic z-score baseline described above - cold-start hold for review, flat-baseline safety, and severity from deviation magnitude - and normalizes each detected issue to an Event(event_type="anomaly.finding") in observation mode via to_event, keyed by detector + metric + window so repeated ticks dedup.
Seasonality: core/detection/seasonal.py (SeasonalAnomalyDetector) handles metrics with a periodic shape so a normal per-phase peak (a Monday-morning traffic spike, a nightly batch job) does not fire against a pooled 24x7 mean. It buckets history by a configured phase (hour_of_day, day_of_week, hour_of_week, or a custom function) and compares the observed sample only against past samples in the same phase. It is a thin wrapper over the base detector - it filters history to the phase and delegates the z-score, cold-start-hold for review, flat-baseline, and event-normalization logic - so the two detectors cannot drift. Per-phase cold-start is independent (a thin Sunday baseline never borrows Monday’s data), the phase is recorded on the detected issue’s window_bucket, and the detected issue is still a observation mode event.
Multivariate fusion: core/detection/composite.py (CompositeAnomalyDetector) is the compound-degradation signal an organization’s on-call reads by hand - a real incident is correlated streams firing together (latency up and error-rate up and saturation high), not one noisy metric. It is a fuser, not a new baseline: it consumes the per-metric AnomalyFinding objects already produced for one resource + window and raises a CompositeAnomalyFinding (event_type="anomaly.composite") only when a configured quorum of them fire. Below quorum it holds for review (a single noisy stream is not a compound anomaly - false-positive suppression); at quorum and above it amplifies (severity escalates with both the breadth of concurrent members and their root-sum-square combined magnitude, so a compound degradation outranks any single member). Duplicate metrics collapse to their strongest occurrence so a re-emitted stream cannot inflate the quorum, a flat-baseline member contributes a fixed weight, and the fusion is deterministic regardless of member order. The composite is still a observation mode detected issue governed by the risk gate - it detects harder, it does not act.

Operational insight recipe catalog

core/detection/insights.py adds a deterministic recipe evaluator for operational conditions that do not need a statistical model. A caller supplies normalized current, previous, and baseline values plus sample counts and last-seen timestamps. The evaluator applies one of ten explicit operators (above, below, delta, percentage change, ratio, absent, or stale) and records the observed value, reference, score, threshold, and explanation in an operational-insight.finding event. Incomplete, non-finite, undersampled, or division-by-zero inputs hold without a detected issue.

The versioned catalog at rule-catalog/operational-insights/catalog.yaml supplies 50 initial recipes:

Infrastructure and telemetry (9): CPU, memory, disk, restart, process, peer-hotspot, freshness, ingestion-volume, and cardinality conditions.
Change and application performance (9): deployment latency, errors, throughput, request errors, tail latency, application performance score, dependency amplification, trace critical path, and span errors.
Data and active checks (9): slow queries, lock waits, consumer lag, dead-letter growth, synthetic availability and latency, log volume, new log patterns, and rare errors.
SLO, alert quality, and ownership (8): fast and slow burn, error budget, alert storms, flapping, stale evaluation, no-data, and missing ownership.
Cost governance (6): daily spend change, budget overrun, unallocated and idle spend, unit cost, and container request waste.
Security, impact, and recovery hygiene (9): critical misconfiguration, excess privilege, sensitive-data growth, runtime threats, reachable vulnerabilities, impacted sessions, certificate expiry, backup freshness, and network retransmission.

Thresholds and metric bindings remain catalog data, so an environment can tune them without changing the evaluator. Every recipe defaults to observation mode, uses a stable key derived from engine, recipe, resource, and window, and re-enters event-ingest for deduplication before trust routing.

core/detection/insight_source.py (OperationalInsightSource) is the runtime bridge to the shared MetricProvider seam. It queries each distinct metric once per resource and window, derives current, previous, and historical baseline values, and evaluates the catalog as one normalized observation. A successful empty query can prove absent; a provider error marks the metric unavailable and suppresses every dependent recipe, so a telemetry outage cannot be mistaken for a workload outage. Stale recipes extend their bounded lookback to twice the stale threshold; if no last-seen sample exists in that window, the recipe holds instead of inferring one.

Distributed trace continuity

The scheduled analyzer can also evaluate one deployment-configured distributed trace topology without treating that topology as an inventory Resource. A topology target declares a stable generic reference and an ordered set of expected hop names. The Azure delivery adapter reads bounded workspace-based Application Insights rows and normalizes only scenario identity, trace identity, hop name, observation time, and immutable evidence references. Deployment values and query credentials remain outside the repository. The 60-second detection bucket is independent from a 900-second evidence lookback. This preserves repeat idempotency and Incident correlation bounds while covering the documented 120-300-second Log Analytics ingestion floor.

The detector compares each completed scenario run with the expected topology:

Continuous: one trace identity contains every expected hop in order. The run records healthy observation evidence and emits no detected issue.
Regenerated context: all expected hops were observed, but no single trace identity covers the topology. The detected issue records the disconnected trace fragments and the exact hop boundary.
Dropped context: one or more expected hops are absent across the whole run. The detected issue records the missing hops without guessing which component removed the context.
Unknown: the source is unavailable, truncated, outside retention, or has no completed run. The tick holds without a healthy result or a detected issue, so missing telemetry cannot prove a workload failure.

Each discontinuity starts in observation mode and publishes one safe-to-retry (idempotent) Event per topology, scenario, and observation window. The Event uses the scenario correlation key, carries bounded immutable evidence references, and re-enters event-ingest. Repeated high-severity detected issues can therefore open one deduplicated Incident through Heimdall’s existing threshold and lifecycle checks. Root-cause analysis can distinguish the observed boundary from candidate instrumentation, collector, or header-propagation causes only when cited telemetry supports that distinction. The detector never restarts a service or changes a gateway policy. A proposed recovery action still passes the normal safety check, human approval when required, the seven safeguards, and independent effect verification.

3. Predictive / Forecasting

Proactive detection: forecast a threshold breach before it happens - the AIOps “predict capacity bottlenecks and service failures” use case - kept deterministic-first.

Method: trend extrapolation on a measured series (linear/seasonal fit) to a configured forecast horizon, raising a detected issue when the projected value crosses a configured threshold. Every forecast carries its horizon and a confidence interval; it is a projection with stated uncertainty - not deterministic truth and not an LLM oracle - and it never grants execution eligibility.
Targets: capacity/quota exhaustion, replication-lag drift toward RPO breach, cost run-rate vs budget, certificate/secret expiry, backup-retention drift. RPO/RTO and FinOps targets are owned by phase-3-integrated-loop.md.
Backtesting before promotion: a forecaster must backtest on historical series (predict known past breaches) and clear an accuracy bar in observation mode before it may leave observation mode.
Drift: forecast error is tracked over time; measured degradation (drift) automatically demotes the forecaster back to observation mode.
Safety: a prediction raises a detected issue (observation mode by default) or a proactive fix PR; it never auto-executes on its own. Acting on a forecast still passes the risk gate and carries all seven safeguards.
Measurement: define lead time = actual_breach_time − finding_time (a valid prediction has positive lead time above an actionable minimum) and score precision/recall, where a true positive is a predicted breach whose actual breach occurs within the horizon. A missed breach is a false negative (guard metric); a poor forecaster stays in observation mode.
Upstream implementation: core/detection/forecast.py (LinearForecastDetector) ships a least-squares linear forecaster - cold-start and weak-fit (low R-squared) inputs hold for review, direction-gated rising/falling breach projection, and a positive lead time (breach ETA) bounded by the horizon. Each forecast normalizes to an Event(event_type="forecast.finding") in observation mode via to_event, keyed by detector + metric + window so repeated ticks dedup; severity scales with imminence (lead / horizon). It shares the MetricSample series type with the anomaly detector (core/detection/series.py).
Prediction-interval band (false-positive suppression): core/detection/forecast_band.py (prediction_band) adds the uncertainty band a point forecast lacks. A noisy series can cross the threshold on the center line yet stay inside normal variation; the band widens with the fitted residual_std and with how far into the future the projection reaches, and a breach is only confident when the pessimistic edge of the interval (the lower edge for a rising breach, the upper edge for a falling one) still crosses at a configured confidence level (0.80-0.99). It is a suppressor, never an amplifier: it can downgrade a point-estimate breach to “not confident” (hold in observation mode / hold for review, protecting the false-positive guard metric), but it never manufactures a breach the point forecast did not predict. A perfect fit (residual_std == 0) collapses the band to the point estimate; an unknown confidence level is rejected rather than silently defaulted.

Forecast validation and outcome closure

A forecast is not evidence of predictive quality until its horizon closes against an observed outcome. FDAI keeps prediction fidelity separate from response effectiveness so a preventive action that avoids a breach does not make a useful forecast look like a false positive.

Immutable prediction envelope. Before publishing a forecast detected issue, record a stable prediction_id, detector and configuration version, target resource and metric, breach predicate, event-time feature cutoff, horizon, projected breach time, point estimate, uncertainty interval, and mode. The envelope is append-only. A later detector version creates a new prediction rather than rewriting the old one.

Outcome closure. A scorer closes the prediction only after horizon_end + telemetry_grace. The grace period is configured from the measured ingestion-delay distribution. Labels use event time, not processing time, and follow these rules:

Observed episode	Prediction label	Treatment
The declared breach occurs within the horizon and no preventive action changed the target	true positive	Measure lead time from detected issue to breach.
No declared breach occurs within the horizon, telemetry is complete, and no preventive action ran	false positive	Count against precision for that exact horizon.
A declared breach occurs without an eligible earlier prediction	false negative	Create the denominator from actual breach episodes, not only emitted forecasts.
A preventive action runs after the prediction and the breach does not occur	intervention-censored	Exclude from forecast precision; score the action in the response ledger.
Telemetry is missing or stale, the resource is deleted, or an excluded maintenance window overlaps	unscorable	Count and report; never coerce to a true negative.

A breach after the declared horizon is not a true positive for that horizon. It remains evidence for horizon selection and can match a separate longer-horizon prediction. Duplicate observations join by the stable prediction and incident keys, so at-least-once delivery cannot score twice.

Two ledgers. The prediction-fidelity ledger stores forecast-to-outcome joins. The response ledger stores the intervention, preconditions, expected effect, observed effect, verification, rollback, SLO recovery, and recurrence window. An intervened episode is never used as an untreated forecast label. For safety-critical actions FDAI does not withhold a proven response to manufacture a control group; use observation mode-only predictions, naturally untreated episodes, matched historical cohorts, or a reviewed stepped rollout for counterfactual evidence.

The response ledger’s first runtime slice is implemented. The control loop emits the strict ResponseOutcome contract after independent effect observation, including expected range, observed value, time window, verification, execution mode, rollback result, target digest, and an explicit scorable marker. The existing scheduled growth job consumes these records under an independent watermark and updates only registered observation mode challenger models. SLO recovery, recurrence closure, matched cohorts, and promotion to quasi_experimental or interventional evidence remain follow-up work and are not inferred from a verified effect alone.

Leakage-safe evaluation. Backtests use rolling-origin time splits and group all events from one incident into one split. Features, topology, maintenance state, and labels are read only as of the prediction cutoff. The incumbent and candidate run over the same frozen replay and then the same live observation mode events; the candidate cannot execute. Report each target and horizon separately, with sample size and confidence intervals, using precision, recall, false alerts per resource-day, PR-AUC, Brier score or calibration error, interval coverage, actionable lead-time distribution, abstention, cold-start, and unscorable rates. Aggregate accuracy alone is not a promotion metric.

Agent choreography. Heimdall owns the forecast detected issue and deterministic outcome closure; Huginn supplies normalized actual observations; Saga records immutable prediction and terminal evidence; Norns analyzes closed case-history cohorts off-path and proposes inert detector/rule candidates; Mimir owns reviewed promotion. Forseti may judge and Thor may act on a detected issue, but neither may edit its prediction label. These agents consume typed events independently and may run in parallel; no direct agent call is part of the scoring path.

Promotion requires a pre-registered minimum number of closed, scorable episodes and observation days, candidate improvement whose confidence interval clears the incumbent, no guard-metric regression, and zero policy escapes. Calibration, recall, interval coverage, or actionable lead time degradation automatically returns the detector to observation mode. The durable episode ledger, event-time outcome join, intervention censoring, transactional publication outbox, and mechanical tick wiring are implemented. Promotion still depends on measured deployment evidence and the authoritative promotion registry.

The PostgreSQL forecast episode store exposes a read-only operational evidence snapshot. Its deterministic reducer reports precision, recall, missed-breach rate, interval coverage, mean and median positive lead time, non-positive lead-time count, abstention rate, explicit outcome counts, and denominator gaps. A breach already present before the detected issue is retained as non-actionable evidence instead of failing the snapshot or inflating lead time. Censored and unscorable outcomes stay visible but do not enter the scorable denominator, and the snapshot carries no execution or promotion authority. Console closure completeness uses only episodes whose closure deadline is due; future episodes do not count as failures, and an empty due cohort is unmeasured. Outcome rows retain scored labels and miss origins from object.forecast-outcome outbox records, never lifecycle closure reasons. Console publication debt counts only unpublished, non-dead-letter rows whose availability time is due. A future retry is not current debt, and the oldest pending time retains the original creation time of the oldest due row instead of resetting its age to a retry time.

4. Root-Cause Analysis

The tier contract, deterministic causal chain, grounded reasoning, knowledge evidence, and read- only projection are owned by Root-Cause Analysis.

Plugging Into the Control Loop

Correlation runs inside event-ingest. Anomaly and forecast detectors are out-of-band producers (e.g. event-driven Functions per app-shape.instructions.md and phase-1 out-of-band detection) that publish detected issues onto the bus; those detected issues re-enter event-ingest to get an idempotency key and dedup, so a flapping detector cannot inject duplicate work. No detector is a new autonomy surface:

telemetry / metrics
  -> anomaly / forecast detectors emit findings ---.               # sections 2-3
  raw events -------------------------------------- +-> event-ingest
                                                       (normalize + dedup + correlate)   # section 1
  -> trust-router -> T0 | T1 | (T2 -> quality-gate)                                       # RCA per tier, section 4
  -> risk-gate -> auto -> executor -> delivery (PR) | HIL | abstain/deny -> audit

A detected issue is a first-class, versioned event type in shared/contracts with a stable idempotency key (e.g. detector-id + metric + window-bucket, or the incident id), so repeated evaluation ticks deduplicate instead of piling up.
Detectors are configuration-driven (baselines, thresholds, horizons, correlation keys, and model bindings are config, not hard-coded), honor observation mode-before-enforce, and every detected issue and decision is audited.

AIOps Alignment

What we adopt from the general AIOps model, and where we intentionally differ:

AIOps capability	Our stance
Incident detection & alerting	Adopt - correlation + anomaly emit detected issues
Root-cause analysis	Adopt - first-class RCA per tier (section 4)
Anomaly detection	Adopt - statistical, explainable (section 2)
Predictive analytics	Adopt - trend + threshold forecast, with uncertainty (section 3)
Alert-noise reduction / fewer false positives	Adopt - correlation + measured FP rate
Less manual work / faster resolution	Adopt - risk-gated auto-fix
Audit trails / compliance	Adopt - append-only audit is already core
ML/NLP as the primary engine	Differ - deterministic-first; models are the 5-10% residual
Opaque / black-box anomaly scoring	Differ - explainable-first; a detected issue records its baseline, deviation, and direction
*Model recommends and* executes**	Differ - execution eligibility is from deterministic verification, not the model
Vendor-platform lock-in	Differ - CSP-neutral; observability platforms are telemetry sources, not the brain

Configuration and Safety

The repository-governed config/detection-governance-policy.json pins detector methods, cold-start floors, forecast target families and horizons, exact correlation keys and windows, backtest promotion thresholds, and change-window treatment. Core loads the policy through core/detection/governance_policy.py with exact-field validation, so an unknown method, duplicate identity, weakened zero-escape guard, or malformed bound fails startup. The current executable binding enforces the forecast-target section: every FDAI_FORECAST_TARGETS_JSON entry names its governed target_kind, and startup rejects a horizon or confidence level that differs from policy and any sample or fit floor that weakens it. The anomaly, correlation, promotion, and change-window sections are reviewed configuration contracts for their existing fixed evaluators; this slice does not claim separate runtime override bindings for them.
Baselines, deviation thresholds, forecast horizons, correlation keys, and model bindings are configuration; a fork overrides them via the DI seams in project-structure.md, never by editing core.
Detectors validate their config at startup and fail closed - a broken detector, an insufficient/cold-start baseline, or stale telemetry makes the detector hold for review rather than emit a false detected issue or auto-act.
Repeated-event Incident policy is startup-bound Runtime Settings: incident.auto_open.enabled (default true), incident.auto_open.min_severity (default medium), incident.repeat_threshold (default 5, range 2-100), incident.repeat_window_seconds (default 300, range 10-86400), incident.security_high_threshold (default 5, range 1-100), incident.security_window_events (default 100, range 1-10000), and incident.alert_rate_per_hour (default 5, range 1-1000). Invalid values fail startup. Severity maps deterministically from critical/high/medium/low/info to SEV1-SEV5; composition does not replace every candidate with a fixed severity. The default admits a persistent medium-severity detector episode only after the repeat and evidence gates succeed. Low and informational episodes remain held unless an installation explicitly lowers the minimum.
Detection detected issues are untrusted input; any LLM use (fuzzy correlation, T2 RCA) passes the quality gate (architecture.instructions.md) and the prompt-injection threat model in security-and-identity.md.
Emit detector metrics - fire rate, false-positive rate, false-negative/missed-breach rate, hold for review and cold-start-suppression counts, forecast lead time, and RCA groundedness - to the KPI dashboard.

Runtime delivery status

The scheduler delivery path publishes canonical, idempotent Events to the configured Event Hubs ingest topic. The analyzer Terraform job invokes fdai.delivery.analyzer_tick_cli, which resolves its targets from the configured list plus the durable inventory projection, runs the reference analyzers against the composed MetricProvider, and publishes one canonical Event per detected issue with a key derived from the resource, the signal, and the detected issue observation’s one-minute publication bucket. The default five-minute analysis window is independent from this retry-stable publication identity. Inventory-backed resolution is read-only and fail-closed: a resource type without a The fdai-incident-evidence-query maintenance entry point reads the durable Incident audit through the service-owned store and returns only transition, distinct-Incident, maximum-member, and kind aggregates for one bounded correlation prefix. It never returns Incident IDs, member IDs, payloads, or database configuration. Inventory-backed resolution is read-only and fail-closed: a resource type without a The fdai-forecast-evidence-query maintenance entry point returns the repeatable-read forecast health snapshot with explicit episode denominators and null rates when no scorable denominator exists. The snapshot is evidence-only and never grants promotion authority. Inventory-backed resolution is read-only and fail-closed: a resource type without a reviewed analyzer mapping is skipped, an observed state fact that is stale, conflicting, partial, or synthetic is skipped with a stable reason, and an unreadable projection raises instead of degrading to the configured list alone, so the Job retries rather than silently narrowing coverage. The resolver accepts the canonical generic state fact from either the legacy flat metadata shape or the current property-keyed collection and never substitutes another property-specific fact. A resource whose metadata collection has no generic state fact follows the identity-and-type-only enumeration path; this read-only selection makes no state claim and grants no authority. A present generic state remains admission-gated and a malformed value remains unusable. A resource projected without any state fact follows the same enumeration path. Discovered targets are bounded and deterministically ordered. Before provider I/O, the resolver holds a discovered target as telemetry_source_unavailable when the currently bound telemetry catalogs do not cover every metric its reference analyzer requires. The candidate remains visible in held coverage and no provider query is sent. Explicitly configured targets retain their declared failure behavior, and an error from a bound provider still fails the tick. The resolver applies the reviewed analyzer Resource types as a store-side filter, then reads a relationship-free bounded window of up to 1,000 supported Resources before applying the configured analyzable-target cap. Unrelated inventory records cannot consume the query window or target slots, and a relationship coverage gap cannot invalidate identity-only target selection. A truncated supported-resource window or incomplete Resource source fails the tick and reports target discovery unavailable rather than presenting partial coverage as healthy. These jobs don’t execute changes; detected issues and due tasks re-enter the shared trust router and safety check. A missing analyzer, provider analysis error, publish failure, or receipt failure keeps a scheduled item retryable and returns a non-zero job result; partial target coverage is never reported as a clean pass. When tracked state and a retry-stable explicit or Container Apps Job execution identity are configured, every completed pass also retains one content-digested runtime:analyzer-tick-receipt: record with target-resolution counts, detected issue publication, trace-continuity outcomes, and readiness. The stable Job execution id is the parent identity, each serial pass has a stable tick ordinal, and the report digest is the attempt identity. An exact retry is a no-op, a changed retry becomes a separate content-addressed attempt, and identical later ticks do not collapse into the first tick’s receipt. Every receipt carries execution_authority: false. Local runs without a stable execution identity do not create an operational receipt. Protected split-service deployment verifies provenance, SBOM, and Core model-material attestations from the digest-pinned GHCR OCI subject, avoiding unrelated public Blob DNS dependencies on the private deployment runner. The protected plan-observability-* and apply-observability-* request family targets one state-only Terraform updater rather than the analyzer resource’s broad prerequisite graph. It accepts only digest-pinned ACR images and one fixed Job/container name. Apply updates that image, verifies authoritative readback, and restores the prior digest when verification fails. The original Container Apps Job resource remains the declarative owner, so the updater converges to the same configured image without taking ownership of any other Job property.

Azure resource create, update, and delete signals flow continuously through the canonical Event Hubs ingress. Huginn owns this real-time discovery ingress and preserves the resource identity, change kind, and bounded properties in the normalized Event. A dedicated projector applies resource, link, and tombstone deltas to the durable inventory overlay in partition order. Complete ARG/ARM reconciliation remains a separate bounded completeness proof, but the target policy runs it adaptively rather than using a fixed six-hour freshness cadence. Durable lag, churn, maximum staleness, request budgets, provider throttling, bounded backoff, and circuit state determine the next action. Heimdall detects stale snapshots, cursor lag, fallback spikes, coverage loss, and provider pressure. A missing, degraded, or stale freshness lookup routes graph-dependent actions to human review. Inventory-backed readiness probes preserve that freshness state instead of asserting discovery success. Heimdall remains an observer. A re-arming no-progress deadline and an absolute ceiling let a stalled source fail without terminating a slow source that continues to produce batches. The complete collection, retention, rollup, and archive contract is defined in Continuous Operational Instance Graph.

Resolved Decisions

Decision	Governed default
Anomaly method by signal class	Stationary reliability and security activity use z-score. Periodic reliability and cost signals use seasonal z-score with an explicit phase.
Forecast family and horizon	Every current target uses the implemented linear trend family. Capacity uses 24 hours, replication lag 1 hour, cost 7 days, and expiry 30 days.
Correlation	Exact `correlation_id` and `resource_ref` keys precede T1. Analyzer detected issue publication uses 60-second retry-stable identities while the trace/repeat window is 300 seconds. Fuzzy T1 requires similarity of at least `0.85` plus two shared evidence fields.
Cold start	Stationary classes require 30 baseline samples, seasonal classes require 10 same-phase samples, and forecasts require 5 samples plus `R-squared >= 0.5`.
Backtesting and promotion	Evaluate weekly after at least 14 observation mode days and 30 scorable episodes. Precision and recall must each be at least `0.8`, 90% interval coverage must remain in `[0.85, 0.95]`, median lead time must be at least 300 seconds, abstention must be at most `0.2`, and policy escapes must remain zero.
Change windows	An exact-scope active window with complete evidence annotates the detected issue and holds Incident promotion. Missing, stale, incomplete, or mismatched window evidence cannot suppress a detected issue.

To learn about	Read
Delivery status and remaining work	Implementation ledger