Observability and Detection

How FDAI turns raw telemetry into detected issues the control loop can act on: event correlation, anomaly detection, predictive / forecasting, and root-cause analysis (RCA). These are the detection signals an AIOps platform is expected to provide - added here without breaking deterministic-first: every signal emits a normalized detected issue that flows through the existing trust-router → tiers → risk-gate → executor → audit path, never a side channel, and nothing auto-executes outside the safety check and the seven safeguards.

Reference: control loop, tiers, and the quality gate in architecture.instructions.md; measurement and guard metrics in goals-and-metrics.md; rule/signal sources in rule-catalog-collection.md; module placement and DI seams in project-structure.md; the prompt-injection threat model in security-and-identity.md. Correlation and out-of-band detection are introduced in phase-1-rule-catalog-t0.md; FinOps cost-anomaly and DR RPO/RTO forecasting land in phase-3-integrated-loop.md. Customer-agnostic; all examples are synthetic.

Design Stance (deterministic-first, not ML-first)

Detection is explainable and evidence-backed first: statistical baselines, thresholds, and correlation rules resolve most signals with no model call. Models (T1 similarity, T2 reasoning) enter only for fuzzy correlation and novel RCA - the same 5-10% budget.
A detection signal is a detected issue, not an action. It is routed and risk-gated like any event; a prediction or anomaly never auto-remediates on its own - it raises a observation mode detected issue or a fix PR that the safety check and human approval govern.
Routine monitoring is not an Incident. A healthy heartbeat, successful probe, or within-threshold sample records observation evidence only. An Incident can open only after a detector emits a bounded, grounded detected issue and IncidentLifecycleWorkflow rechecks the allowed agent principal, correlation keys, reason, and member-event evidence.
A repeated-event burst is an anomaly, not automatic Incident authority. Heimdall always records the bounded anomaly, but it can hand off an Incident candidate only when the normalized Event declares incident_correlation=correlate, carries a non-empty correlation id and evidence key, and meets the configured minimum severity. Every event in one repeated-event burst belongs to the same non-empty correlation episode; events from independent episodes never satisfy one another’s threshold or interrupt their independent accumulation. The burst severity is the most severe recorded value in that bounded window, not the value on whichever Event arrived last. Every Event that satisfies the threshold contributes its stable evidence key to the candidate and the resulting Incident member set. Events marked incident_correlation=none, including inventory and discovery changes, never open an Incident. The default automatic-open minimum is high; an unclassified burst remains medium and stays an anomaly. If anomaly publication or the lifecycle handoff fails, Heimdall retains that bounded episode window and retries only when the next matching Event arrives; it does not create an unbounded background retry loop. The handoff reports accepted or held, and Heimdall records those outcomes separately so a policy hold is never counted as a successful Incident candidate. A more severe recurrence for an open Incident raises its severity through an append-only incident.severity row; a recurrence never lowers severity, replay reconstructs the same monotonic result, and the committed escalation emits a deduplicated A2 lifecycle notice. Direct candidate text and evidence keys are capped at 512 characters, and one candidate carries at most 100 evidence keys; oversized input is held before lifecycle or audit writes.
Heimdall bounds retained repeated-event episodes globally and per resource. A correlation flood from one resource evicts only that resource’s oldest episode before it can displace another resource’s partially accumulated evidence.
New detectors ship in observation mode and are promoted per the observation mode→enforce rule; their accuracy and false-positive rate are measured against the Phase 0 baseline.

Frozen configuration baseline checks

Configuration drift is a T0 (deterministic rules) detected issue. A reviewed actual snapshot is frozen as intended state and is never replaced automatically by a later observation. The human-readable DOCX and canonical JSON baseline carry the same version, scope, creation time, and document digest. Generation and validation also verify that every displayed resource, attribute, evidence gap, topology link, exception, and unknown item in canonical JSON appears in the paired DOCX. A matching file digest alone does not establish cross-format equivalence.

core/detection/configuration_drift.py canonicalizes resources, topology links, and comparable attributes, then reports added, removed, changed, unchanged, unknown, or unauthorized.
A partial snapshot cannot prove removal. Missing resources, links, or attributes remain blocked until an authoritative source supplies complete evidence.
The configured baseline version, SHA-256 digest, and scope are server-owned. A caller cannot select another target through tool arguments.
An immutable baseline registry can hold candidate, active, superseded, and archived versions for multiple scopes. It allows one active version per scope. Active and replay-pinned sources are selected by server composition, never by conversational input, and the registry exposes no mutation API.
The shipped observation source is delivery/configuration_drift.py, whose JsonFileConfigurationObservationSource reads one bounded scope-pinned JSON document for development and evidence replay. A live Azure Resource Graph observation adapter is not implemented, so a current-state drift answer is only as current as the supplied document. The resource-group contains link and provider-id redaction helpers described here exist in delivery/azure/arg_projection.py and today serve the inventory and discovery adapters, not the drift path.
Knowledge retrieval explains and cites the reviewed document. It does not decide the drift. If Knowledge is unavailable, the deterministic report remains valid and the citation status stays blocked rather than being reported as supported. Each citation identity includes the exact baseline version and full DOCX SHA-256 digest so a reused filename cannot alias another document. Exact metadata lookup takes precedence; a bounded deterministic lexical fallback ranks chunks only within that pinned document and returns no result for an unrelated query. Provider exceptions emit a structured warning containing the exception type and pinned baseline identity, but never the exception message or chunk content.
The read-only capability reports mutation, approval, mitigation, and unsupported-claim counts. Each remains zero for a configuration check.
The public bind_configuration_drift composition helper installs this one server-pinned A0 capability through the immutable capability runtime. It does not add an ActionType, executor identity, schedule authority, or caller-selected scope.
Every fresh run records baseline-load, observation, comparison, Knowledge, and total latency plus resource and detected issue counts. Current observations are not reused through a TTL cache because a cached snapshot cannot satisfy a current-state question. The receipt rejects stage latencies that exceed total elapsed time beyond floating-point timer tolerance.
A pure review reducer accepts three idempotent run receipts for one pinned baseline. Only three verified runs can produce an inert weekly schedule proposal. Any blocked or unsafe run pauses the campaign, and the reducer never creates a scheduler task directly. A revisioned StateStore adapter persists campaign progress with atomic state-and-audit create and compare-and-set advance.
Before campaign advance, an immutable StateStore report ledger records the full detected issues, citations, safety counters, and measured performance under the campaign and run identity. Its strict codec supports restart replay, duplicate content is a no-op, and identity reuse with different evidence is blocked.
A ready campaign submits an inert Automation Blueprint. Independent review and the authenticated scheduler command remain mandatory before a observation mode weekly event exists. Configuration drift does not call the scheduler store or executor directly.

1. Event Correlation

A stage in event-ingest, immediately after normalize + deduplicate (see project-structure.md and phase-1-rule-catalog-t0.md): group related raw events into a single incident so downstream tiers reason about one thing, not a storm.

Deterministic-first: correlate by shared keys (resource id, deployment id, trace/correlation id, causal parent) within a bounded time window, using rules; fall back to T1 embedding similarity only for fuzzy grouping.
Grouping, not causation: correlation only asserts events belong together; a shared window can be coincidental. Assigning the cause is RCA’s job (section 4), never correlation’s.
Windowing and late arrival: the correlation window is configured per signal class; a late/out-of-order event matching an open incident’s keys is attached to it (or, past the window, opens a linked follow-on incident) - events are never silently dropped, and the per-resource ordering from architecture.instructions.md is preserved.
Idempotent grouping: the incident id is derived deterministically from the correlation keys, so reprocessing the same members yields the same incident regardless of arrival order.
Noise reduction: a burst of alerts from one root event collapses to one incident. This is reported as a measured noise-reduction ratio (incidents ÷ raw alerts), not an asserted gain, and no data is lost - members stay linked in the audit.
Output: one correlated incident event carrying its member event ids and a stable idempotency key; ordering/idempotency keys are preserved.
Lifecycle boundary: a correlation id is an investigation key, not proof that an Incident exists. IncidentRegistry owns the lifecycle record. Audit-only local fixtures remain available to Audit and Trace but are excluded from the operational Incident roster.
Operational-work policy: normalized Events declare incident_correlation. The default correlate preserves incident grouping. Discovery, inventory, scheduler, and workflow-control producers set none, retain correlation_id for trace and audit, and derive no Incident ID.
Upstream implementation: core/event_ingest/correlator.py (EventCorrelator) derives the incident anchor deterministically from an event’s correlation-id (or resource ref) plus a time-window bucket via incident_id_for; a burst sharing a key in one window collapses to one incident and a new window opens a linked follow-on. An event with incident_correlation=none or no anchor is reported correlated=False (never dropped). The keys feed IncidentRegistry.open, which accumulates membership idempotently.

2. Anomaly Detection

Generalizes the existing FinOps cost-anomaly hook (see phase-3-integrated-loop.md) to any metric stream - performance, reliability, security, and cost.

Method: statistical baselines (rolling and/or seasonal, with the seasonality window as config) and deviation thresholds (e.g. z-score or robust percentile bands), computed per signal class. Deterministic and explainable; the baseline, the deviation magnitude, and its direction (over/under) are recorded so a human can see why it fired.
Cold-start: a detector without enough baseline history to be reliable holds for review (stays in observation mode and emits no detected issue) rather than firing on a thin baseline; the cold-start suppression is counted as a metric, not hidden.
Categories: detected issues normalize to the canonical category enum (security | reliability | cost | config_drift | compliance) shared with the rule catalog - performance signals (latency/error-rate/saturation) and replication lag map to reliability, unusual access patterns to security, spend run-rate to cost. Severity derives from deviation magnitude.
Change-aware suppression: anomalies coincident with an in-flight change/maintenance window are correlated with the originating change event and suppressed or annotated, so a deploy does not manufacture false positives.
False-positive and false-negative control: debounce/settling windows plus a measured false-positive rate and false-negative (missed-anomaly) rate that a new detector must not regress - both map to guard metrics in goals-and-metrics.md.
Output: an anomaly detected issue that re-enters event-ingest (for an idempotency key and dedup) and then the trust router like any event.
Upstream implementation: core/detection/anomaly.py (MetricAnomalyDetector) ships the deterministic z-score baseline described above - cold-start hold for review, flat-baseline safety, and severity from deviation magnitude - and normalizes each detected issue to an Event(event_type="anomaly.finding") in observation mode via to_event, keyed by detector + metric + window so repeated ticks dedup.
Seasonality: core/detection/seasonal.py (SeasonalAnomalyDetector) handles metrics with a periodic shape so a normal per-phase peak (a Monday-morning traffic spike, a nightly batch job) does not fire against a pooled 24x7 mean. It buckets history by a configured phase (hour_of_day, day_of_week, hour_of_week, or a custom function) and compares the observed sample only against past samples in the same phase. It is a thin wrapper over the base detector - it filters history to the phase and delegates the z-score, cold-start-hold for review, flat-baseline, and event-normalization logic - so the two detectors cannot drift. Per-phase cold-start is independent (a thin Sunday baseline never borrows Monday’s data), the phase is recorded on the detected issue’s window_bucket, and the detected issue is still a observation mode event.
Multivariate fusion: core/detection/composite.py (CompositeAnomalyDetector) is the compound-degradation signal an organization’s on-call reads by hand - a real incident is correlated streams firing together (latency up and error-rate up and saturation high), not one noisy metric. It is a fuser, not a new baseline: it consumes the per-metric AnomalyFinding objects already produced for one resource + window and raises a CompositeAnomalyFinding (event_type="anomaly.composite") only when a configured quorum of them fire. Below quorum it holds for review (a single noisy stream is not a compound anomaly - false-positive suppression); at quorum and above it amplifies (severity escalates with both the breadth of concurrent members and their root-sum-square combined magnitude, so a compound degradation outranks any single member). Duplicate metrics collapse to their strongest occurrence so a re-emitted stream cannot inflate the quorum, a flat-baseline member contributes a fixed weight, and the fusion is deterministic regardless of member order. The composite is still a observation mode detected issue governed by the risk gate - it detects harder, it does not act.

Operational insight recipe catalog

core/detection/insights.py adds a deterministic recipe evaluator for operational conditions that do not need a statistical model. A caller supplies normalized current, previous, and baseline values plus sample counts and last-seen timestamps. The evaluator applies one of ten explicit operators (above, below, delta, percentage change, ratio, absent, or stale) and records the observed value, reference, score, threshold, and explanation in an operational-insight.finding event. Incomplete, non-finite, undersampled, or division-by-zero inputs hold without a detected issue.

The versioned catalog at rule-catalog/operational-insights/catalog.yaml supplies 50 initial recipes:

Infrastructure and telemetry (9): CPU, memory, disk, restart, process, peer-hotspot, freshness, ingestion-volume, and cardinality conditions.
Change and application performance (9): deployment latency, errors, throughput, request errors, tail latency, application performance score, dependency amplification, trace critical path, and span errors.
Data and active checks (9): slow queries, lock waits, consumer lag, dead-letter growth, synthetic availability and latency, log volume, new log patterns, and rare errors.
SLO, alert quality, and ownership (8): fast and slow burn, error budget, alert storms, flapping, stale evaluation, no-data, and missing ownership.
Cost governance (6): daily spend change, budget overrun, unallocated and idle spend, unit cost, and container request waste.
Security, impact, and recovery hygiene (9): critical misconfiguration, excess privilege, sensitive-data growth, runtime threats, reachable vulnerabilities, impacted sessions, certificate expiry, backup freshness, and network retransmission.

Thresholds and metric bindings remain catalog data, so an environment can tune them without changing the evaluator. Every recipe defaults to observation mode, uses a stable key derived from engine, recipe, resource, and window, and re-enters event-ingest for deduplication before trust routing.

core/detection/insight_source.py (OperationalInsightSource) is the runtime bridge to the shared MetricProvider seam. It queries each distinct metric once per resource and window, derives current, previous, and historical baseline values, and evaluates the catalog as one normalized observation. A successful empty query can prove absent; a provider error marks the metric unavailable and suppresses every dependent recipe, so a telemetry outage cannot be mistaken for a workload outage. Stale recipes extend their bounded lookback to twice the stale threshold; if no last-seen sample exists in that window, the recipe holds instead of inferring one.

Distributed trace continuity

The scheduled analyzer can also evaluate one deployment-configured distributed trace topology without treating that topology as an inventory Resource. A topology target declares a stable generic reference and an ordered set of expected hop names. The Azure delivery adapter reads bounded workspace-based Application Insights rows and normalizes only scenario identity, trace identity, hop name, observation time, and immutable evidence references. Deployment values and query credentials remain outside the repository.

The detector compares each completed scenario run with the expected topology:

Continuous: one trace identity contains every expected hop in order. The run records healthy observation evidence and emits no detected issue.
Regenerated context: all expected hops were observed, but no single trace identity covers the topology. The detected issue records the disconnected trace fragments and the exact hop boundary.
Dropped context: one or more expected hops are absent across the whole run. The detected issue records the missing hops without guessing which component removed the context.
Unknown: the source is unavailable, truncated, outside retention, or has no completed run. The tick holds without a healthy result or a detected issue, so missing telemetry cannot prove a workload failure.

Each discontinuity starts in observation mode and publishes one safe-to-retry (idempotent) Event per topology, scenario, and observation window. The Event uses the scenario correlation key, carries bounded immutable evidence references, and re-enters event-ingest. Repeated high-severity detected issues can therefore open one deduplicated Incident through Heimdall’s existing threshold and lifecycle checks. Root-cause analysis can distinguish the observed boundary from candidate instrumentation, collector, or header-propagation causes only when cited telemetry supports that distinction. The detector never restarts a service or changes a gateway policy. A proposed recovery action still passes the normal safety check, human approval when required, the seven safeguards, and independent effect verification.

3. Predictive / Forecasting

Proactive detection: forecast a threshold breach before it happens - the AIOps “predict capacity bottlenecks and service failures” use case - kept deterministic-first.

Method: trend extrapolation on a measured series (linear/seasonal fit) to a configured forecast horizon, raising a detected issue when the projected value crosses a configured threshold. Every forecast carries its horizon and a confidence interval; it is a projection with stated uncertainty - not deterministic truth and not an LLM oracle - and it never grants execution eligibility.
Targets: capacity/quota exhaustion, replication-lag drift toward RPO breach, cost run-rate vs budget, certificate/secret expiry, backup-retention drift. RPO/RTO and FinOps targets are owned by phase-3-integrated-loop.md.
Backtesting before promotion: a forecaster must backtest on historical series (predict known past breaches) and clear an accuracy bar in observation mode before it may leave observation mode.
Drift: forecast error is tracked over time; measured degradation (drift) automatically demotes the forecaster back to observation mode.
Safety: a prediction raises a detected issue (observation mode by default) or a proactive fix PR; it never auto-executes on its own. Acting on a forecast still passes the risk gate and carries all seven safeguards.
Measurement: define lead time = actual_breach_time − finding_time (a valid prediction has positive lead time above an actionable minimum) and score precision/recall, where a true positive is a predicted breach whose actual breach occurs within the horizon. A missed breach is a false negative (guard metric); a poor forecaster stays in observation mode.
Upstream implementation: core/detection/forecast.py (LinearForecastDetector) ships a least-squares linear forecaster - cold-start and weak-fit (low R-squared) inputs hold for review, direction-gated rising/falling breach projection, and a positive lead time (breach ETA) bounded by the horizon. Each forecast normalizes to an Event(event_type="forecast.finding") in observation mode via to_event, keyed by detector + metric + window so repeated ticks dedup; severity scales with imminence (lead / horizon). It shares the MetricSample series type with the anomaly detector (core/detection/series.py).
Prediction-interval band (false-positive suppression): core/detection/forecast_band.py (prediction_band) adds the uncertainty band a point forecast lacks. A noisy series can cross the threshold on the center line yet stay inside normal variation; the band widens with the fitted residual_std and with how far into the future the projection reaches, and a breach is only confident when the pessimistic edge of the interval (the lower edge for a rising breach, the upper edge for a falling one) still crosses at a configured confidence level (0.80-0.99). It is a suppressor, never an amplifier: it can downgrade a point-estimate breach to “not confident” (hold in observation mode / hold for review, protecting the false-positive guard metric), but it never manufactures a breach the point forecast did not predict. A perfect fit (residual_std == 0) collapses the band to the point estimate; an unknown confidence level is rejected rather than silently defaulted.

Forecast validation and outcome closure

A forecast is not evidence of predictive quality until its horizon closes against an observed outcome. FDAI keeps prediction fidelity separate from response effectiveness so a preventive action that avoids a breach does not make a useful forecast look like a false positive.

Immutable prediction envelope. Before publishing a forecast detected issue, record a stable prediction_id, detector and configuration version, target resource and metric, breach predicate, event-time feature cutoff, horizon, projected breach time, point estimate, uncertainty interval, and mode. The envelope is append-only. A later detector version creates a new prediction rather than rewriting the old one.

Outcome closure. A scorer closes the prediction only after horizon_end + telemetry_grace. The grace period is configured from the measured ingestion-delay distribution. Labels use event time, not processing time, and follow these rules:

Observed episode	Prediction label	Treatment
The declared breach occurs within the horizon and no preventive action changed the target	true positive	Measure lead time from detected issue to breach.
No declared breach occurs within the horizon, telemetry is complete, and no preventive action ran	false positive	Count against precision for that exact horizon.
A declared breach occurs without an eligible earlier prediction	false negative	Create the denominator from actual breach episodes, not only emitted forecasts.
A preventive action runs after the prediction and the breach does not occur	intervention-censored	Exclude from forecast precision; score the action in the response ledger.
Telemetry is missing or stale, the resource is deleted, or an excluded maintenance window overlaps	unscorable	Count and report; never coerce to a true negative.

A breach after the declared horizon is not a true positive for that horizon. It remains evidence for horizon selection and can match a separate longer-horizon prediction. Duplicate observations join by the stable prediction and incident keys, so at-least-once delivery cannot score twice.

Two ledgers. The prediction-fidelity ledger stores forecast-to-outcome joins. The response ledger stores the intervention, preconditions, expected effect, observed effect, verification, rollback, SLO recovery, and recurrence window. An intervened episode is never used as an untreated forecast label. For safety-critical actions FDAI does not withhold a proven response to manufacture a control group; use observation mode-only predictions, naturally untreated episodes, matched historical cohorts, or a reviewed stepped rollout for counterfactual evidence.

The response ledger’s first runtime slice is implemented. The control loop emits the strict ResponseOutcome contract after independent effect observation, including expected range, observed value, time window, verification, execution mode, rollback result, target digest, and an explicit scorable marker. The existing scheduled growth job consumes these records under an independent watermark and updates only registered observation mode challenger models. SLO recovery, recurrence closure, matched cohorts, and promotion to quasi_experimental or interventional evidence remain follow-up work and are not inferred from a verified effect alone.

Leakage-safe evaluation. Backtests use rolling-origin time splits and group all events from one incident into one split. Features, topology, maintenance state, and labels are read only as of the prediction cutoff. The incumbent and candidate run over the same frozen replay and then the same live observation mode events; the candidate cannot execute. Report each target and horizon separately, with sample size and confidence intervals, using precision, recall, false alerts per resource-day, PR-AUC, Brier score or calibration error, interval coverage, actionable lead-time distribution, abstention, cold-start, and unscorable rates. Aggregate accuracy alone is not a promotion metric.

Agent choreography. Heimdall owns the forecast detected issue and deterministic outcome closure; Huginn supplies normalized actual observations; Saga records immutable prediction and terminal evidence; Norns analyzes closed case-history cohorts off-path and proposes inert detector/rule candidates; Mimir owns reviewed promotion. Forseti may judge and Thor may act on a detected issue, but neither may edit its prediction label. These agents consume typed events independently and may run in parallel; no direct agent call is part of the scoring path.

Promotion requires a pre-registered minimum number of closed, scorable episodes and observation days, candidate improvement whose confidence interval clears the incumbent, no guard-metric regression, and zero policy escapes. Calibration, recall, interval coverage, or actionable lead time degradation automatically returns the detector to observation mode. The durable episode ledger, event-time outcome join, intervention censoring, transactional publication outbox, and mechanical tick wiring are implemented. Promotion still depends on measured deployment evidence and the authoritative promotion registry.

4. Root-Cause Analysis

Make RCA a first-class output of the tiers instead of an implicit side effect.

Tier	RCA role
T0	direct cause: the matched rule/policy names the violated control and its fix
T1	correlation cause: either (a) match the incident to a prior resolved incident and reuse its identified root cause + learned action (with provenance and re-verification), or (b) reconstruct a deterministic causal chain from the incident’s own correlated events - identify the closest antecedent change / mutation that preceded the failure within a bounded window on a related resource (the “a deploy went out, then the error rate rose” chain)
T2	reasoning cause: for novel/ambiguous incidents, produce a grounded root-cause hypothesis that cites evidence (rules, correlated events, telemetry, free-form operator documents) and passes the quality gate

RCA output is a hypothesis with citations, not an authoritative decision; execution eligibility is still granted by deterministic verification (verifier + policy re-check), never by the RCA text or a forecast alone.
Telemetry and correlated events feeding T2 RCA are untrusted input and may carry prompt injection; per security-and-identity.md the verifier and policy re-check are authoritative over any model text.
T1 reuse of a prior resolved incident’s root cause must re-verify that the prior cause and its learned action still apply (with provenance), and any resulting action runs what-if before the safety check - a stale learned action is never replayed blindly.
An RCA that cannot be grounded holds for review and routes to human approval.
The correlated incident (section 1) is the RCA input, so RCA reasons over one incident, not a storm of duplicates.
Upstream implementation: core/rca/ ships the RCA contract (RootCauseHypothesis + Citation), the deterministic T0 cause (t0_root_cause, grounded on the matched rule with confidence 1.0 and its fix), and the evidence check gate (enforce_grounding, which holds for review to human approval on any ungrounded or below-confidence hypothesis). The T2 reasoner is the RcaReasoner Protocol seam - a fork plugs a mixed-model, RAG-grounded producer (via core/quality_gate) behind it. Upstream ships core/rca/llm.py (LlmRcaReasoner + the RcaModel seam) whose deterministic parser refuses a malformed answer, a fabricated citation (prompt injection), or an ungrounded answer - the model proposes, the parser and evidence check gate decide. The Azure T2 binding is delivery/azure/llm/rca_model.py (AzureOpenAIRcaModel), an RcaModel adapter that calls Azure OpenAI over its managed-identity token and returns raw JSON for the upstream parser to validate. The composition root binds it from the t2.rca capability in resolved-models.json (bind_azure_llm_bindings), symmetric to the Critic and Judge bindings - a missing capability or prompt leaves LlmBindings.rca_reasoner = None so T2 RCA stays dark and only T0 RCA runs. __main__ injects the resulting RcaCoordinator (and the EventCorrelator) into the ControlLoop. Its output still passes the evidence check gate and the safety check verifier, never executing on the model’s prose alone. The RcaCoordinator orchestrates all three tiers - T0, T1 correlation-reuse (a prior resolved incident’s cause, abstaining when it is stale against current evidence), and T2 (a citation outside the supplied evidence is refused as fabricated). It is wired into the ControlLoop, which appends one deterministic T0 rca.hypothesis audit entry per detected issue, carrying the correlated incident_id (from EventCorrelator, section 1) so an incident’s detected issues tie together - the “why”, never a new execution path. When a T2 reasoner is wired, a novel (T0 no-match) case additionally gets a grounded T2 rca.hypothesis (or an hold for review), reasoner-gated so a deployment without an LLM emits no T2 noise.
Free-form knowledge leg: core/rca/knowledge_evidence.py (KnowledgeEvidenceGatherer) is the RCA consumer of the Knowledge Base ingestion seam (shared/providers/knowledge.py KnowledgeSource + EmbeddingKnowledgeSource / PgvectorKnowledgeSource). When bound, the RcaCoordinator’s T2 convenience wrappers search the operator’s ingested documents (runbooks, architecture notes, resource plans) for chunks relevant to the incident summary and add each as a CitationKind.KNOWLEDGE candidate - so a document an operator uploads is actually referenced when T2 forms a hypothesis. Fail-safe (an unbound source, empty index, or provider outage contributes nothing and the gate holds for review) and secret-safe (a citation ref is the opaque knowledge:<source_ref>#<chunk_id> handle, never the chunk body). The reasoner still cannot cite a chunk outside this vouched-for set, and the evidence check gate + verifier remain authoritative.
T1 causal chain (deterministic): core/rca/causal_chain.py (CausalChainAnalyzer, driven by core/rca/t1.py’s t1_causal_chain) is the model-free form of T1 correlation (b). Given the incident’s correlated events (each carrying a timestamp, a generic resource_ref, an is_change marker, and an optional change_kind), it reconstructs the most probable multi-hop causal chain ending at the failure - root change -> symptom -> ... -> failure - not merely the single closest antecedent. The root MUST be a change (a mutation can cause; a symptom only propagates), so a window of pure symptoms with no antecedent change holds for review (returns None, deferring to T2). Reconstruction is dependency-aware: when a resource-dependency graph is supplied, a change on a resource the failure depends on (directly, or transitively within a bounded depth) outranks an unrelated one, and once a graph is given an unrelated resource cannot link at all; with no graph the engine stays permissive (any correlated resource may link - the cross-resource default). same_resource_only restricts every hop to the failing resource. Confidence is a weakest-link aggregate over the chain’s hops (each hop weighted by temporal proximity, relationship strength, and change-kind), ambiguity-discounted when several distinct roots explain the failure about equally well, and bounded to the T1 band (0.35-0.85)
- a temporal antecedent is a strong hint, never T0-style certainty. Strict temporal precedence makes the event set a DAG, so the chain is deterministic (the same events always yield the same chain) and cites every event in it; it passes the evidence check gate and the safety check verifier before anything acts. RcaCoordinator.analyze_t1_causal_chain is the grounded entry point. Live wiring: the ControlLoop feeds it each matched incident’s members through the IncidentMemberSource seam (core/rca/member_source.py; a fork’s adapter marks which members are changes) and appends one observation mode rca.hypothesis (tier t1) per event, bounded by the configured causal_chain_window and an optional resource-dependency graph. The hypothesis retains a transport-safe causal_chain (root/failure ids, ambiguity, and ordered hop evidence), and the control loop writes that structure into the append-only audit entry instead of collapsing it into prose. The upstream reference implementation DeploymentHistoryMemberSource (core/rca/deployment_member_source.py) bridges a real DeploymentHistoryProvider (e.g. the Azure Resource Graph adapter) plus an incident-record lookup into the antecedent is_change=True events, so a fork gets live change-history-driven chains without writing the source. Absent a source, T1 causal-chain RCA stays dark and only T0 (and T2, when wired) RCA runs (backward-compatible).
Read-only console surface: the observation mode rca.hypothesis audit entries are projected into a first-class History > RCA operator-console panel (GET /rca?correlation=<id>, pure projection in services/operator-service/src/fdai_operator_service/rca_projection.py). Given an incident correlation_id it renders the tiered hypotheses, their citations, the structured T1 causal chain when recorded, evidence check state (an held for review hypothesis shows as “insufficient evidence check -> human approval”, never a confident cause), and the linked response plan (decision / action / mode / rollback) composed from the same correlated audit stream. The surface is strictly read-only and adds no new source of truth - see operator-console.md.

Plugging Into the Control Loop

Correlation runs inside event-ingest. Anomaly and forecast detectors are out-of-band producers (e.g. event-driven Functions per app-shape.instructions.md and phase-1 out-of-band detection) that publish detected issues onto the bus; those detected issues re-enter event-ingest to get an idempotency key and dedup, so a flapping detector cannot inject duplicate work. No detector is a new autonomy surface:

telemetry / metrics
  -> anomaly / forecast detectors emit findings ---.               # sections 2-3
  raw events -------------------------------------- +-> event-ingest
                                                       (normalize + dedup + correlate)   # section 1
  -> trust-router -> T0 | T1 | (T2 -> quality-gate)                                       # RCA per tier, section 4
  -> risk-gate -> auto -> executor -> delivery (PR) | HIL | abstain/deny -> audit

A detected issue is a first-class, versioned event type in shared/contracts with a stable idempotency key (e.g. detector-id + metric + window-bucket, or the incident id), so repeated evaluation ticks deduplicate instead of piling up.
Detectors are configuration-driven (baselines, thresholds, horizons, correlation keys, and model bindings are config, not hard-coded), honor observation mode-before-enforce, and every detected issue and decision is audited.

AIOps Alignment

What we adopt from the general AIOps model, and where we intentionally differ:

AIOps capability	Our stance
Incident detection & alerting	Adopt - correlation + anomaly emit detected issues
Root-cause analysis	Adopt - first-class RCA per tier (section 4)
Anomaly detection	Adopt - statistical, explainable (section 2)
Predictive analytics	Adopt - trend + threshold forecast, with uncertainty (section 3)
Alert-noise reduction / fewer false positives	Adopt - correlation + measured FP rate
Less manual work / faster resolution	Adopt - risk-gated auto-fix
Audit trails / compliance	Adopt - append-only audit is already core
ML/NLP as the primary engine	Differ - deterministic-first; models are the 5-10% residual
Opaque / black-box anomaly scoring	Differ - explainable-first; a detected issue records its baseline, deviation, and direction
*Model recommends and* executes**	Differ - execution eligibility is from deterministic verification, not the model
Vendor-platform lock-in	Differ - CSP-neutral; observability platforms are telemetry sources, not the brain

Configuration and Safety

Baselines, deviation thresholds, forecast horizons, correlation keys, and model bindings are configuration; a fork overrides them via the DI seams in project-structure.md, never by editing core.
Detectors validate their config at startup and fail closed - a broken detector, an insufficient/cold-start baseline, or stale telemetry makes the detector hold for review rather than emit a false detected issue or auto-act.
Repeated-event Incident policy is startup-bound Runtime Settings: incident.auto_open.enabled (default true), incident.auto_open.min_severity (default high), incident.repeat_threshold (default 5, range 2-100), and incident.repeat_window_seconds (default 300, range 10-86400). Invalid values fail startup. Severity maps deterministically from critical/high/medium/low/info to SEV1-SEV5; composition does not replace every candidate with a fixed severity.
Detection detected issues are untrusted input; any LLM use (fuzzy correlation, T2 RCA) passes the quality gate (architecture.instructions.md) and the prompt-injection threat model in security-and-identity.md.
Emit detector metrics - fire rate, false-positive rate, false-negative/missed-breach rate, hold for review and cold-start-suppression counts, forecast lead time, and RCA groundedness - to the KPI dashboard.

Runtime delivery status

The scheduler delivery path publishes canonical, idempotent Events to the configured Event Hubs ingest topic. The analyzer Terraform job invokes fdai.delivery.analyzer_tick_cli, which resolves its targets from the configured list plus the durable inventory projection, runs the reference analyzers against the composed MetricProvider, and publishes one canonical Event per detected issue with a key derived from the resource, the signal, and the tick window. Inventory-backed resolution is read-only and fail-closed: a resource type without a reviewed analyzer mapping is skipped, an observed state fact that is stale, conflicting, partial, or synthetic is skipped with a stable reason, and an unreadable projection raises instead of degrading to the configured list alone, so the Job retries rather than silently narrowing coverage. A resource projected without a state fact carries identity and type only, which is what target selection needs, so it stays eligible. Discovered targets are bounded and deterministically ordered. These jobs don’t execute changes; detected issues and due tasks re-enter the shared trust router and safety check. Publish failure keeps a scheduled item retryable and returns a non-zero job result.

Azure resource create, update, and delete signals flow continuously through the canonical Event Hubs ingress. Huginn owns this real-time discovery ingress and preserves the resource identity, change kind, and bounded properties in the normalized Event. A dedicated projector applies resource, link, and tombstone deltas to the durable inventory overlay in partition order. The Inventory job separately promotes a complete ARG/ARM reconciliation snapshot every six hours by default and retires overlay entries covered by the new generation. Heimdall detects stale snapshots, cursor lag, fallback spikes, and coverage loss. A missing, degraded, or stale freshness lookup routes graph-dependent actions to human review. Inventory-backed readiness probes preserve that freshness state instead of asserting discovery success. Heimdall remains an observer. The Inventory job checks durable attempt state every 10 minutes, runs the normal six-hour scan only when due, and retries a newer failed or abandoned attempt on the next tick without granting job-start authority to the core runtime.

Implementation status

Implementation scope

Area	State	Evidence	Notes
Event correlation	implemented	`services/core-control-plane/src/fdai/core/event_ingest/correlator.py`; `services/core-control-plane/tests/core/event_ingest/test_correlator.py`	Deterministic grouping, episode bounds, and stable incident identity are covered by focused tests.
Anomaly and composite detection	implemented	`services/core-control-plane/src/fdai/core/detection/anomaly.py`; `seasonal.py`; `composite.py`; focused `tests/core/detection/test_*.py`	Cold start, flat baselines, quorum, duplicate collapse, and explainable scores fail closed.
Forecasting and outcome closure	implemented	`services/core-control-plane/src/fdai/core/detection/forecast.py`; `forecast_outcome.py`; `forecast_closure.py`; focused forecast tests	Prediction, censoring, and closure contracts are implemented. Promotion still requires measured deployment evidence.
Configuration drift	implemented	`services/core-control-plane/src/fdai/core/detection/configuration_drift.py`; `configuration_drift_service.py`; focused configuration-drift tests	Frozen baselines, deterministic comparison, review, and reporting remain evidence-only.
Live configuration observation	not-started	`services/core-control-plane/src/fdai/delivery/configuration_drift.py` ships only `JsonFileConfigurationObservationSource`; no adapter exists under `delivery/azure/`	The `ConfigurationObservationSource` seam is defined and bound in composition, but `bind_configuration_drift` is called only from tests, never from runtime bootstrap. Drift cannot answer a current-state question about live Azure until an adapter exists.
Scheduled analyzer delivery	implemented	`services/core-control-plane/src/fdai/delivery/analyzer_tick.py`; `analyzer_tick_cli.py`; `infra/modules/compute/container-apps/analyzer_tick_job.tf`; `services/core-control-plane/tests/delivery/test_analyzer_tick.py`	The configured entry point exists and publishes one canonical, window-keyed Event per detected issue. A publish failure is reported and exits non-zero so the Job retries. Deployed-runtime evidence is still outstanding.
Inventory-backed target resolution	implemented	`services/core-control-plane/src/fdai/delivery/analyzer_targets.py`; `services/core-control-plane/src/fdai/core/investigation/analyzers.py`; `services/core-control-plane/tests/delivery/test_analyzer_targets.py`; `tests/integration/infra/test_detection_readiness.py`	One tick analyzes the configured targets plus every eligible `Resource` in the durable inventory projection. Unmapped types, unusable or stale observed state facts, and a failed projection read all fail closed. Deployed-runtime evidence is still outstanding.
Distributed trace continuity	implemented	`core/detection/trace_continuity.py`; `delivery/azure/trace_continuity.py`; `delivery/trace_continuity_tick.py`; analyzer Job binding; focused detector, source, tick, Incident, human approval, and Terraform checks (`55 passed`)	Deterministic evaluation, strict bounded Azure normalization, observation mode Event publication, and repeated-detected issue Incident creation are implemented. Live Azure detection, approval, and recovery evidence remain open in issue #142.
Governed operational accuracy	in-progress	Runtime delivery status; Open decisions	Runtime precision, recall, interval coverage, lead time, and false-positive evidence remain deployment work.

Implementation history

Date	State	Change	Evidence	Remaining
2026-08-14	in-progress	Adopted the implementation ledger without reconstructing earlier provenance and corrected the analyzer delivery claim to match the current tree.	`current change`; current source and focused tests listed in the scope table.	Restore analyzer delivery and retain governed accuracy evidence.
2026-08-15	implemented	Added the analyzer tick runner and the `fdai.delivery.analyzer_tick_cli` entry point the Terraform job configures, publishing one canonical window-keyed Event per detected issue with reported publish failures.	`current change`; `services/core-control-plane/src/fdai/delivery/analyzer_tick.py`; `pytest services/core-control-plane/tests/delivery/test_analyzer_tick.py` (10 passed).	Retain deployed accuracy evidence; target resolution is the configured list only.
2026-08-16	not-started	Corrected three claims this document made about code that does not exist as described. The frozen-baseline bullet named `delivery/azure/configuration_drift.py` and an Azure Resource Graph query; no such module exists and the only shipped observation source is file-backed. Live configuration observation is now a separate `not-started` scope row rather than being implied by the `implemented` drift row.	`current change`; `find services -name "configuration_drift.py"` returns only `core/detection/` and `delivery/configuration_drift.py`, whose module docstring reads “File-backed baseline sources”; `grep -rn bind_configuration_drift` shows runtime bootstrap never calls it.	Build the Azure observation adapter, or record a decision that drift stays evidence-replay-only.
2026-08-16	not-applicable	Repointed two stale references: the RCA projection moved to `services/operator-service/src/fdai_operator_service/rca_projection.py`, and the shared category list omitted `compliance` and used a hyphen where `Category` uses `config_drift`.	`current change`; `find services -name "rca_projection*.py"`; `services/core-control-plane/src/fdai/shared/contracts/models/enums.py` `Category` has five members.	None; both are now exact.
2026-08-16	implemented	Resolved analyzer-tick targets from the durable inventory projection in addition to the configured list. A reviewed neutral resource-type map selects the analyzer kind, configured targets keep priority, discovered targets are bounded and deterministically ordered, and unmapped types, unusable or stale observed state facts, and a failed projection read fail closed instead of narrowing coverage silently.	`current change`; `services/core-control-plane/src/fdai/delivery/analyzer_targets.py`; `pytest services/core-control-plane/tests/delivery/test_analyzer_targets.py services/core-control-plane/tests/delivery/test_analyzer_tick.py` (24 passed); strict mypy and Ruff passed the changed files.	Record deployed-runtime evidence that an inventory-discovered resource joins a live tick.
2026-08-16	implemented	Hardened inventory-backed resolution after review. The discovered bound now stops one row below the durable store’s own query limit so the documented maximum cannot raise inside the projection read, `FDAI_ANALYZER_MAX_DISCOVERED_TARGETS` is rejected at parse time with the environment key named, a state fact whose evidence cutoff is not timezone-aware is skipped as unusable instead of raising, truncation is reported only when a target was actually withheld, and the deployed job binds the `FDAI_INVENTORY_DSN` key the CLI reads. Determinism is now claimed only for an untruncated projection.	`current change`; `services/core-control-plane/src/fdai/delivery/analyzer_targets.py`; `pytest services/core-control-plane/tests/delivery/test_analyzer_targets.py services/core-control-plane/tests/delivery/test_analyzer_tick.py services/core-control-plane/tests/delivery/test_analyzer_tick_routed.py` (30 passed); `pytest tests/integration/infra/test_detection_readiness.py` (3 passed).	Record deployed-runtime evidence that an inventory-discovered resource joins a live tick.
2026-08-17	in-progress	Accepted the deterministic distributed-trace continuity design after rejecting an inventory-resource mapping that would have misrepresented a trace topology as a managed Resource.	`current change`; this document; issue #142.	Implement and focus-test the source, detector, shared analyzer Job binding, and governed Event path, then retain live `preserve`, `regenerate`, `drop`, approval, and recovery evidence.
2026-08-17	implemented	Implemented the distributed-trace continuity detector, strict workspace-based Application Insights source, shared analyzer Job runner and configuration, and repeated-detected issue Incident handoff. The KQL uses the documented `Id`, `OperationId`, `Properties`, and `TimeGenerated` columns.	`current change`; focused behavior and human approval checks passed 55 cases; strict mypy and task-scoped Ruff passed; `terraform -chdir=infra validate` succeeded.	Deploy the exact validated revision to the observation lab and retain live `preserve`, `regenerate`, `drop`, approval, and recovery evidence before advancing this scope to `validated`.

Remaining work

The analyzer entry point the Terraform job configures exists, publishes canonical window-keyed Events, and reports publish failures with a non-zero result, proven by services/core-control-plane/tests/delivery/test_analyzer_tick.py.
Analyzer targets resolve from the configured list plus the durable inventory projection through a reviewed neutral resource-type map, and unmapped types, unusable or stale observed state facts, and a failed projection read fail closed, proven by services/core-control-plane/tests/delivery/test_analyzer_targets.py.
Implement a live ConfigurationObservationSource for Azure and bind it from runtime bootstrap, evidenced by a focused adapter test and a bootstrap binding test; until then the Live configuration observation scope row stays not-started and a drift answer is only as current as the supplied document.
Record deployment evidence for detector precision, recall, missed breaches, interval coverage, forecast lead time, and abstention rates.
Record deployed-runtime evidence that an inventory-discovered resource joins a live analyzer tick without a deployment edit, and retain the resulting tick report.
Complete issue #142 with focused checks and live Azure evidence that preserve stays healthy, regenerate and drop produce evidence-backed detected issues, repeated detected issues open one Incident, and the recovery path reaches human approval or a fully safeguarded action before verified closure.
Resolve the signal-class methods, baseline history, and promotion thresholds in Open decisions and encode them in governed configuration.

Open Decisions

Anomaly method per signal class (z-score vs robust percentile vs seasonal decomposition).
Forecast model family and default horizons per target (capacity, lag, cost, expiry).
Correlation key set and time-window defaults; when to escalate fuzzy correlation to T1.
Cold-start policy: minimum baseline history per signal class before a detector may fire.
Backtesting cadence and the accuracy bar a forecaster must clear to leave observation mode.
Change-window suppression: how anomalies are correlated with in-flight change events.