Operating and Verification

Use this guide to verify that FDAI is alive, correct, and behaving from a freshly provisioned deployment onward. This document is self-observability: how the system reports on itself. It is distinct from observability-and-detection.md, which is what the system detects about the environment it watches. Presentation / dashboard layout is out of scope for this document.

Complements deploy-and-onboard.md (provisioning) and startup-and-lifecycle.md (bootstrap). Azure focus: non-Azure providers are TBD (see Always-On Rules).

Self-Health Signals

Signals a healthy deployment MUST emit continuously. Every signal maps 1:1 to an alert rule (see the alert-routing contract).

Signal	Purpose	Failure mode caught
Liveness probe (per container)	container process alive	crash loop
Readiness probe (per container)	dependencies reachable	boot without Kafka broker / Key Vault reference / DB
Adapter healthcheck (per provider adapter)	Kafka broker reachable (Event Hubs :9093), Key Vault reference resolvable, Diagnostic-Settings forwarders healthy, catalog loaded in OPA, T2 model endpoints reachable	silent dependency drop
Event lag (ingest to first tier decision)	per-tier latency	ingress backpressure
DLQ depth (per queue / topic)	dead-letter accumulation	poison message, consumer failure
Cold-start rate + duration	scale-to-zero warm-up cost	deadline misses (routes to human approval)
Verifier failure rate	T2 verifier hold for review / fail rate	drift in verifier accuracy
Mixed-model disagreement rate	cross-check disagreement	model degradation
Rollback rate	actions later reverted	miscalibrated rules or actions
Override rate	override create / modify per rule	poor-fit rules (feeds the discovery loop)
Discovery loop pass rate	candidate → quality gate pass %	loop drift
Kill-switch state	on / off	contained emergency posture
Canary result	synthetic loop round-trip	silent ingress death
Time since last successful canary	staleness	monitor of the monitor

Local source bindings

The production composition reads liveness from the opened health listener, readiness from the same guarded state used by /ready, and kill-switch state from its authoritative StateStore key. The configuration field identifies the validated runtime-policy values consumed at bootstrap, not a claim that pending restart-required settings have taken effect. The version field identifies the installed Core distribution, not a deployment or source-revision attestation. These process-local observations do not establish health for other replicas or services.

Cold-start duration measures this Core process from runtime entry to its first observed guarded ready state after bootstrap composition. It uses monotonic elapsed time and retains the original UTC observation time and an opaque process-instance source id. Before that observation, after shutdown, or without the lifecycle binding, the signal is unavailable. Recollection never renews the observation’s freshness. This single process sample supplies neither a fleet cold-start rate nor a complete installation-wide window.

Process-owned first-decision lag

The normal Core stage publisher can supply event_lag from accepted ingestion to the first completed T0, T1 or T2 verification, measured with one process-local monotonic clock. Selecting a route, broker acceptance and synthetic canaries are not tier decisions. Only the trusted runtime’s canary-no-op route excludes a matched ingress; a caller’s event-type label cannot hide lag. The source retains the largest recent completed latency and every unresolved event’s current age, so a fast completion cannot hide a stalled sibling. Bounded-state overflow, incomplete stage pairs and clock regression make the source unavailable rather than healthy. A missing binding, no observation, stale completed samples and shutdown remain explicit. No stage detail, provider identity or event body is retained.

This is a process-owned measurement, not broker backlog, other replicas’ latency, verifier accuracy or operational-effect evidence. The reviewed design rejects route timing as a proxy and never resets the oldest start on duplicate ingress. Ordinary pipeline publication is unchanged.

Consumed-canary age

The age source reads the latest committed successful control_loop.canary audit from the configured Core store in append-sequence order, with one row and an 8 KiB entry-transfer ceiling. A dedicated partial index keeps that read bounded; the oldest-first migration reader is not a substitute. It validates the canonical actor, event, correlation, outcome, timestamps, latency and selected-row hash before measuring elapsed seconds. Missing, malformed, future or unreachable evidence stays unavailable, with no older-row fallback.

Age uses the original UTC recorded_at, captured before audit append, so it can conservatively overstate time since completion. Exact Decimal construction, independent of caller precision, preserves the 600-second policy boundary. The observation time is the current age measurement, not a renewed canary success. Repeated reads therefore increase age and eventually alert instead of refreshing old evidence into Healthy. This source does not establish the latest attempt result, round-trip latency, complete audit history or execution authority. The age source alone cannot qualify canary_result or its state field.

Initial option	Critique	Source boundary
Use publisher or startup broker success	Neither proves consumption through the canary path	Read the independently persisted consumed-canary audit
Read an oldest-first audit prefix	A prefix cannot identify the latest successful canary	Use an indexed latest-success query
Reuse an old success as a fresh boolean	It can hide a stalled canary indefinitely	Measure increasing age from the original timestamp; keep latest-attempt status separate

Consumed-attempt lifecycle and the scheduling boundary

Core records an append-only control_loop.canary.attempt audit with outcome started before publishing canary stages, followed by failure or cancelled under that same action kind, or the existing control_loop.canary success audit. Both carry the same Core-generated attempt_id, original event identity and started_at; the store assigns canonical sequence numbers. These records describe consumption, not scheduler execution or publish acceptance. Terminal persistence is bounded. If it fails, cancellation repeats, or the process exits, the durable start remains unresolved; it is not a successful completion. Failure records retain only the exception class, never the exception message or provider payload. An uncertain terminal write is not followed by a contradictory outcome, and failure of the post-commit stage notification does not retract a persisted success. Keeping lifecycle records under a separate action kind preserves the deployment verifier’s single-success-row contract.

This does not complete the latest scheduled-result source. The publisher accepts an explicit FDAI_CANARY_SLOT or one retry-stable platform execution identity described below; it never derives identity from the current clock. A job that never publishes still leaves no consumed event. The controller UID or Container Apps execution name identifies one actual execution, not the latest due schedule or a missing execution. Deriving a five-minute slot from Core’s clock cannot supply that independent witness or distinguish a cancelled publisher from a missing execution.

Without an independent scheduler witness, canary_result and the canary operational-state field remain unavailable. A recent success or these lifecycle records alone cannot qualify them. No publisher database permission is added.

Independently witnessed AKS result

The source combines independent native scheduling metadata and canonical consumption, never publisher acceptance. The critique and resulting boundaries are:

Boundary	Source contract
Native witness	Read the configured CronJob and metadata-only Job list. Its explicit UTC schedule, Kubernetes 1.32+ scheduled-timestamp annotation and current controller UID must identify exactly one latest-due Job. Missing, duplicate, suspended, malformed or truncated observations remain unavailable; older Jobs or controller generations cannot substitute.
Access and transfer	A namespace-local Role grants only named-CronJob get and Job list, without workload mutation, Secret read, executor or publisher database permissions. Limit each response to 256 KiB and the complete list to 16 Jobs, without pagination or retries.
Canonical sequence	Read at most 64 audit rows plus one truncation witness for that exact execution. Validate identity, hashes, increasing sequence, nondecreasing time, and each unique start/matching terminal. Unresolved or overlapping retries cannot borrow an earlier success.
Result and state	Preserve original observation time. Confirmed consumption failure is nonpassing; cancellation has no numeric result and an explicit unavailable canary_cancelled state. Unsupported sources, including unqualified Container Apps witnesses, remain unavailable, not inferred from execution names.
Critique and currentness	Re-read CronJob generation around the Job list and independent scheduler witnesses around the database read. After the final await, reject a changed occurrence or the exclusive next-slot deadline. Replayed collection times and clock regression cannot replay Healthy. Result and state share one bounded collection, including failure, without independent provider retries.
Qualification	Existing freshness, read authorization, approval, promotion and execution boundaries remain unchanged. Local checks are source implementation, not installed-runtime qualification or a live drill.

Retry-stable publisher identity

An exact nonempty FDAI_CANARY_SLOT continues to take precedence for explicitly identified runs. Otherwise the AKS canary uses FDAI_CANARY_JOB_UID, injected through the Downward API from metadata.labels['batch.kubernetes.io/controller-uid'], as the aks-job:<uid> slot. Retry Pods share their controller Job UID even when delayed beyond a minute or five-minute boundary. The protected CronJob digest includes this exact field reference, not its eventual runtime value. No Kubernetes API read, database access or additional role is required.

Using a Pod UID would split retries; rounding the current clock would split delayed retries or merge independent executions. Neither is a fallback. Container Apps uses its automatically supplied CONTAINER_APP_JOB_EXECUTION_NAME as aca-execution:<name>, not the job definition name or replica name. Microsoft Learn’s built-in Jobs variables identify this variable as the execution name (document dated 2026-03-31). No template environment override or additional permission is needed. The canary accepts an exact 1-86-character lowercase alphanumeric/hyphen token, with alphanumeric ends, to preserve its 100-character slot bound.

Without an explicit slot, configuring both platform identity keys is ambiguous and fails before provider access, even if one value is empty or malformed. A missing or malformed sole identity also fails; the publisher never selects another identity or a current-clock fallback. An explicit nonempty FDAI_CANARY_SLOT still wins over platform metadata. Execution identity repairs retry deduplication only. It does not witness a missing scheduled execution or make result/state available.

The catalog’s unqualified adapter_healthcheck remains the configured-adapter aggregate. A source id cannot narrow that aggregate to t1.embedding; without complete authoritative adapter coverage the catalog signal remains unavailable. The earlier single-adapter binding was withdrawn after a two-configured-adapter counterexample produced a false global Healthy. Actual model observations are now exposed separately in typed model_provider_scope and model_provider_registration siblings of the existing response’s unchanged snapshot. The startup registration preserves every configured model role, including unobserved roles. The existing Operator reader verifies the independent registration, exact role/binding set, parent snapshot digest, time, coverage, and freshness before returning this optional sub-scope. Missing registration or any missing/unknown role keeps scoped health unknown; no model sub-scope can substantiate global adapter Healthy. This is not replica or installation-wide coverage. The producers observe normal embedding and explicitly role-bound cross-check SDK invocations. Overlapping invocations form one cohort, closed only when every invocation settles. Success requires the existing HTTP and finite-vector or final finite-JSON checks; a fully known cohort containing any HTTP/response failure is unhealthy. Cancellation and local-only failures in identity, prompt budget, tools, gateway observation or post-call bookkeeping remain unknown, even if a peer succeeds. Observation adds no probe, retry, rate or inferred denominator. Cross-check roles are composition-owned, independent of the optional prompt composer. A latency pool preserves every configured member in its binding digest and validates each member’s identity and time before using the oldest observation. An unobserved, unsupported, stale or wrong-role member cannot be skipped for a selected-only Healthy. Duplicate role sources are diagnosed and held, not resolved by last-wins registration. Only the latest completed cohort observation is retained. Any in-flight invocation, missing observation, closed client, or expired catalog freshness bound keeps the source unavailable. A newly bound adapter starts unobserved and cannot reuse the old instance’s results. API-contract success is not semantic quality or readiness. The original OperationalStateSnapshotV1 schema and digest are unchanged; the optional scoped evidence uses a separate typed contract and digest.

Source collection runs at half the shortest catalog freshness bound, independently of expensive startup probe refreshes. It never invokes models, changes a kill switch, or executes an action. Missing, malformed, or unreachable source records remain unavailable with bounded reason codes; an existing stale or future-dated observation retains its original freshness decision. Source failure codes include source_missing, source_malformed, source_unreachable, source_incomplete, source_stale, and source_from_future. Provider error text never becomes a projection reason.

Mixed-model comparison observations

The unit is one fully settled Core cross-check round, not a model vote or operational episode. The numerator counts rounds with at least one exact disagreement; the denominator counts fully observed rounds. Only the existing Core comparison result supplies this observation. No comparison, model call or retry is added. Incomplete, invalid, drifted or cancelled rounds make the window unavailable instead of becoming disagreement or success. The actual registered primary and secondary SDK source bindings are required; local-fake and human-approval fallback models cannot supply this rate. Only built-in native SDKs and pools composed entirely of those SDKs qualify automatically. Borrowed descriptors, wrappers and subclasses cannot substitute for that producer lineage; their gate behavior is unchanged, but this built-in source remains unavailable. Each round pins the exact qualified model tuple, and an in-flight round also holds an earlier healthy window. One complete policy-bounded process window uses the existing 10% threshold. Warmup, idle gaps, unknown current results, ambiguous binding, clock faults, restart and shutdown hold the source. Original observation time is retained. This is neither model accuracy, action success nor an independently measured policy effect, and it grants no execution or promotion authority.

Deterministic T2 verifier observations

The default Core quality-gate factory wraps its actual RuleBasedVerifier through the unchanged VerifierPolicy seam. Completed calls returning False or None form the numerator; all known completed verifier calls form the denominator. One complete fixed observation window equals the catalog freshness bound, currently 300 seconds. The existing 5% threshold is unchanged. Warmup, no calls, an unknown exception, ambiguous binding, clock failure and shutdown remain unavailable. A current unknown result also holds an earlier healthy window. Recollection preserves the last actual observation time. Process-local counters restart empty, while the existing monitor persists its sanitized snapshot. This measures neither verifier accuracy nor action or recovery success; retries are calls, not independent operating episodes. No additional model call is made.

Other bindings remain explicitly unavailable until their owning source can supply the required observation. Broker queue depth still needs broker observations; first-decision lag uses only the bound Core process above. cold-start rate needs a complete lifecycle window and denominator; rollback, override, and discovery rates need time-bounded numerators and denominators. Complete adapter health needs the registered dependency checks. Canary result/state needs the independent AKS witness and complete sequence above; the age reader alone does not provide it. The oldest-first audit migration reader cannot establish the latest canary. Rule-effect, override, discovery, and break-glass fields still need their owning projections.

Remaining source boundaries

Up to nine of 15 signal slots and four of eight fields have concrete, wired source implementations on eligible AKS installations; without a qualified scheduler witness the prior eight/three ceiling remains. Comparison observations require the native producer bindings above and a complete known window; source implementation is not installed qualification or a promise of Healthy. The indexed latest-success age reader does not establish the latest scheduled attempt or complete canary state. Six other signals and four fields remain unavailable on the qualified path; missing observations or denominators are never zero failures or a successful result. Publication counters do not prove broker depth, startup duration does not prove fleet cold-start rate, and promotion records do not prove executed rollback or per-rule override rates.

Implementation status

Runtime health, startup readiness, transition telemetry, and the synthetic canary path are implemented and covered by focused tests. The deployment workflow now requires the exact consumed canary audit row and a 30-second numeric latency budget after publisher completion. A retained live receipt, complete alerting, fixture replay, drills, and stabilization automation remain incomplete, so no area is claimed as fully operationally validated.

Implementation scope

Area	State	Evidence	Notes
Retry-stable Container Apps canary publisher identity	implemented	Official Microsoft Learn built-in Jobs variables (2026-03-31); canary_cli.py; test_canary_cli.py; focused offline canary cohort: 83 passed	Uses the automatically supplied execution name, not job definition or replica name. Exact names are bounded and namespaced; two platform keys fail closed without an explicit slot. No API, permission or Terraform change is required.
Retry-stable AKS canary publisher identity	implemented	canary_cli.py; aks_workload_jobs.py; aks_job_execution.py; AKS workload Terraform; focused offline canary/deployment cohort: 80 passed	Explicit slots retain precedence; otherwise the Downward API supplies controller Job UID. The protected digest binds its exact field reference. Missing or malformed identity fails before provider access; no current-clock or unverified Container Apps fallback exists.
Core consumed-attempt lifecycle	implemented	_canary.py; test_canary_lifecycle.py; test_postgres_canary.py; focused local canary cohort: 62 passed	Durable start before stages, bounded failure/cancellation persistence, separate canonical success, and no contradictory outcome after an uncertain write. These are consumption records, not a scheduler witness.
Latest scheduled canary result and typed state	implemented	delivery/kubernetes_canary.py; delivery/canary_sequence.py; delivery/self_health_canary_result.py; indexed PostgreSQL sequence; default readiness binding; scoped AKS Role; 153 focused checks including actual isolated PostgreSQL	Source implementation for an independently witnessed AKS occurrence, with missing, malformed, stale, truncated, failed, cancelled, overlapping, changed-generation, replayed and over-deadline observations held or explicitly nonpassing. Unqualified sources stay unavailable; this is not installed qualification or whole-H01 completion.
Scheduled creation intent replay	implemented	creation.py; shared schedule-store duplicate error; PostgreSQL adapter; creation/blueprint/replay checks: 108 passed, creation line/branch coverage 100%	Idempotent registration preserves run progress and stored kind normalization without accepting changed JSON/policy intent. Four new isolated PostgreSQL cases and a reviewed materialization checkpoint failure are covered. This does not register daily jobs or settle H02 operating decisions.
Actual deterministic T2 verifier rate	implemented	delivery/self_health_verifier.py; default gate/readiness/resource ownership; 20 focused cases, 100% branch-inclusive source coverage	Complete policy-bounded process windows, exact returned outcomes, no invented denominator or learned outcome, unchanged gate decisions, typed persisted readback and shutdown. No deployment or operational quality improvement is claimed.
Stabilization scheduler numeric input integrity	implemented	#2254; shared schedule model, compatibility tools and actual creation-command tests; 110 focused checks, model/tool branch-inclusive coverage 92.62%/97.63%	Invalid non-finite intervals cannot be stored or replace a valid task; valid cadence and normal due dispatch remain unchanged. This hardens a prerequisite, not daily-job registration or stabilization qualification.
Runtime /live and /ready endpoints plus startup-readiness evaluation, persistence, transitions, and guarded refresh	implemented	runtime/health.py, runtime/readiness.py, test_readiness.py, and test_startup_coordinator.py	Process-critical failure closes readiness; degraded capabilities lower authority without raising it on recovery.
Bounded transition spans and metrics with secure OTLP configuration	implemented	transitions.py and test_transition_telemetry.py	Stable allowlisted attributes prevent payloads and provider errors from becoming telemetry labels.
Synthetic canary publisher, canonical consumer, no-op audit path, and scheduled Container Apps Job	implemented	canary_cli.py, _canary.py, canary_job.tf, and focused canary tests	The canary is isolated from T0/T1/T2, risk, execution, incident, and learning paths.
Post-deploy promotion smoke	in-progress	deploy-dev.yml; scripts/deployment/azure/verify_canary_audit.py; focused verifier tests	The workflow gives the publisher a deterministic slot, then requires one exact Core audit row within 30 seconds. The verifier rejects ambiguous, malformed, unsettled, future-dated, or over-budget rows without exposing database credentials. A retained live receipt, fixture replay, kill-switch cycle, and human-approval dry-run remain open.
Complete self-health signal policy and routing map	implemented	config/self-health-signals.yaml; core/readiness/self_health.py; production startup binding; focused self-health and readiness tests	All 15 required signals map to one metric, threshold direction, freshness bound, generic owner lane, notification route, and fail-closed fallback. Missing coverage or route drift blocks startup. Decisions grant no execution authority.
Consumed-canary age source	implemented	#2250; delivery/self_health_canary.py; delivery/persistence/postgres_canary.py; concurrent partial-index Core migration; 99 focused checks including four actual isolated PostgreSQL cases	Latest successful consumed audit, selected-row hash/time/identity checks, exact 600-second and 8192/8193-byte boundaries, indexed plan beyond a 512-row oldest prefix, default readiness composition and non-destructive index rollback are locally tested. Caller Decimal precision cannot round away a threshold breach. This is conservative age, not latest-attempt result or operational qualification.
Complete self-health exporter registry, alert delivery, and operational drills	in-progress	Complete signal policy; exact 15-signal and eight-field exporter registry; runtime sources; typed model-role registration/scope; durable operational-state snapshot; authenticated Operator read route	Up to nine signals/four fields on eligible AKS; otherwise eight/three. Six signals/four fields, additional native scheduler witnesses and installed delivery/drill receipts remain open. Event lag is process-owned; comparison requires the qualified native group and full known window. Age alone is not a canary result and model API observations are not aggregate adapter health.
Correlation-based audit investigation and unified version/configuration exposure	implemented	rule_fire_trace.py, audit_rca.py, core/readiness/self_health_runtime.py, and authenticated GET /operations/runtime-state	One sanitized contract now exposes version, configuration, rule effect, override, discovery, canary, kill-switch, break-glass, and all self-health states. Each field includes source, freshness, and an explicit unavailable state; the route grants no approval or execution authority.
Pre-launch latency measurement and frozen-scenario replay	implemented	latency_budget.py, baseline_run.py, and focused tests	The primitives produce bounded measurements and release-gate results; an external load generator and deployment-specific budgets remain operator inputs.
Live Azure read-investigation scenarios	in-progress	Azure Read Investigations	Four bounded read-only scenarios passed, while guest-event matching, an actual provider 429, and cross-service parity receipts remain open release evidence.
ActionType operator-runbook schema and coverage	implemented	check-action-runbooks.py, incident-mitigation-and-rollback.md, and test_check_action_runbooks.py	Every shipped ActionType matches exactly one generic upstream runbook whose declared precondition, procedure, verification, rollback, and audit sections exist. Deployment-specific commands remain fork-owned.
Grounded recurrent-query input	implemented	#2271 / PR #2273; post_turn_procedure.py; norns_recurrent_queries.py; 571 focused regression passes including four isolated PostgreSQL scenarios	Typed executed human-query shapes reach Norns through Operator validation and Bragi metadata transport. Durable distinct-principal counting feeds only the existing inert Mimir input; raw-body consent and current publication gates remain unchanged.
Post-launch stabilization composition	in-progress	core/scheduler/, measurement primitives and the registered recurrent-query input	An accepted configuration-review blueprint now preserves its approved campaign, baseline, scope, runs, cron, and timezone through materialization. Daily health, drift and deployment-baseline jobs remain unregistered. Installed cohort/transport, operating policy and complete stabilization open/demotion/exit evidence remain open; this prerequisite does not complete H02.

Implementation history

Date	State	Change	Evidence	Remaining
2026-10-21	implemented	Bound the independently observed AKS latest due occurrence to a complete canonical consumption sequence and one shared result/state observation. Added namespace-only metadata reads, explicit UTC protected templates, non-destructive concurrent index and replay/ABA/post-await deadline rejection.	current change; 153 focused source, actual loopback PostgreSQL, real producer/index/rollback, runtime persistence and deployment-contract checks passed; nine changed source modules passed mypy; locked signed-provider Terraform formatting/validation passed with no backend or Azure operation. Official Kubernetes CronJob and metadata-only API contracts were checked.	Additional scheduler witnesses, installed readback, complete H01 sources/delivery/drills and operational effects remain unqualified. Missing inputs never become a passing result; cancellation has no numeric measurement. No live Kubernetes/Azure/model request, deployment, promotion or execution-authority increase.
2026-10-20	implemented	Corrected the preceding AKS-only publisher change’s Container Apps scheduling regression using the documented built-in execution name. Explicit slots retain priority; ambiguous dual-platform metadata fails before provider access.	Current change; official Microsoft Learn built-in Jobs variables dated 2026-03-31; eight pre-repair ACA/ambiguity regressions failed; 83 focused offline cases, strict mypy on source/tests, and Ruff passed.	Supersedes the prior row’s Container Apps identity gap only. Independent latest-scheduled-attempt observation, bounded canonical sequence projection and full result/state remain open under #2251. No live or deployment evidence is claimed.
2026-10-20	implemented	Repaired minute-boundary retry identity changes by binding scheduled AKS canaries to the controller Job UID through the Downward API and protected template contract. Explicit FDAI_CANARY_SLOT inputs are unchanged; unbound scheduled publishers fail safely.	Current change; CLI, scheduled-job builder, protected template, Terraform wiring and regression tests; five pre-repair cases failed, then the focused offline canary/deployment cohort passed 80 tests. Strict mypy passed for all three touched source modules plus CLI tests; Terraform formatting passed.	Independent observation of the latest scheduled attempt, bounded canonical audit-sequence projection and full result/state remain open under #2251. Container Apps has no verified execution-identity binding here. No live or deployment evidence is claimed.
2026-10-20	in-progress	Added canonical Core consumed-attempt starts and bounded failure/cancellation records while preserving the existing single-success audit contract and age-only runtime binding.	Current change; _canary.py, test_canary_lifecycle.py, test_postgres_canary.py, and test_canary_cli.py; four regressions failed before repair, then 62 focused offline/isolated-PostgreSQL cases passed.	#2251 still needs pinned schedule/source and independently witnessed retry-stable slots plus a bounded canonical audit-sequence projection. No live or deployment evidence is claimed.
2026-10-20	implemented	Critique round 13 reproduced a normal event borrowing the canary type label and hiding a six-second stall behind a zero-latency sibling. Removed label-based exclusion: only the actual trusted canary no-op route with observed ingress is excluded. Added the real Core canary path and unmatched-route regressions.	Current change; failing-before-fix exact assertion observed 0 instead of 6 seconds. Original stage publication and no-authority contracts are unchanged; exact final focused results are retained on #2286.	This is source correctness, not fleet health, live canary acceptance or completion of the remaining H01 source and qualification gates.
2026-10-20	implemented	Bound actual Core ingestion and first T0/T1/T2 verification stages to one process-owned event-lag source, shared with health collection and resource shutdown. Preserved transport calls and explicit stalled/incomplete states without model or provider probes.	Current change; #2286; 67 focused source/composition/health/Core-stage tests passed; new source has 100% line and branch coverage; six-source strict mypy passed. Twelve distinct bounded review hypotheses cover actual decision versus routing, oldest start, concurrent stall, duplicate completion, staleness, missing/reordered pairs, provenance, canary exclusion, bounded memory, clock regression, shutdown, and unchanged publication/no authority.	Eight signal slots/three fields have concrete sources; seven signals/five fields, installed delivery/drills and operational outcomes remain open. No whole-H01, fleet-health or functional-readiness completion is claimed.
2026-10-19	implemented	Closed an intent-loss defect in the approved configuration-review schedule path: candidate persistence, audit and materialization now retain the exact timezone and pinned campaign/baseline/scope/run payload without exposing payload contents in audit.	current change; 26 focused unit passes, two actual isolated PostgreSQL passes, Ruff, strict mypy, and a 12-perspective hardening review.	No schedule was registered. Approved installation inputs, daily health and deployment-baseline paths, operating policy, transport, and stabilization lifecycle evidence remain open.
2026-10-19	implemented	Bound the actual Core exact cross-check result to a native-producer-qualified logical-round disagreement source, with pending/unknown holds and shared verifier window mechanics. Preserved original model calls, outputs, policies and resource ownership.	Current change; 303 source/Core checks passed with 100% branch-inclusive coverage across comparison helper and both window sources; 529 runtime/safety/SDK checks passed. Three 12-control critique rounds repaired two Medium detected issues: reuse of a qualified observer with another model tuple, and fake direct/pool producers borrowing real SDK descriptors. Exact existing 10% policy, actual gate/bootstrap/persisted contract and shutdown are tested.	Seven implemented signal bindings/three fields, not seven operationally qualified observations. Eight other signals/five fields, complete installation/delivery/drill evidence and measured outcomes remain open. No live model call, deployment or activation.
2026-10-19	implemented	Hardened the actual schedule-creation prerequisite for daily registration: stable definitions replay after execution/restart, typed duplicate races converge, payloads are detached and validated, and policy drift stays denied.	current change; initial 13 counterexamples; 108 focused checks including four new actual PostgreSQL cases; creation coverage 100% including branches; strict mypy.	Daily health/drift/deployment-baseline registration, approved baseline/policy and stabilization qualification remain open.
2026-10-19	implemented	Added actual normal cross-check SDK and complete latency-pool observations to the existing typed model sub-scope. Composition assigns explicit roles without requiring a prompt composer; readiness keeps every registered obligation and diagnoses duplicates.	Current change; 211 focused SDK/composition/Reader checks passed; three SDK modules: 93.82% branch-inclusive coverage. Three 12-control critique rounds reproduced and repaired future-member masking and cancellation during final bookkeeping after a known HTTP failure; 38 new SDK/pool cases pass.	Catalog coverage stays six signals/three fields, with nine signals/five fields unavailable. Other model roles and replicas need actual sources; full H01, installed qualification and operational effects are not established.
2026-10-19	implemented	Reconciled the operating owner with the grounded recurrent-query source added in this change. The earlier unregistered-signal wording is superseded for this metadata-only input, not for daily jobs or operational qualification.	#2271 / PR #2273; typed plan receipt, actual Operator/Bragi/Norns/Mimir path, 571 focused regressions and four actual isolated PostgreSQL scenarios.	Retain daily job registration, actual installed cohort/transport, approved operating decisions and stabilization lifecycle receipts. No whole-H02 Ready claim.
2026-10-19	implemented	Bound the actual default T2 verifier to a complete-window health source without changing its returned decision or invoking another model. Current source coverage is six signals/three fields, not the withdrawn receipt-only six/four interpretation.	Current change; delivery/self_health_verifier.py; tests/delivery/test_self_health_verifier.py: 20 passed, 100% branch-inclusive source coverage. Direct critique reproduced and repaired healthy-window reuse after an unknown result and a registry-identity mismatch.	Nine signals/five fields, installed delivery/drill receipts, and measured operational outcomes remain open. Counter state is intentionally process-local; restart returns to warmup rather than reusing a prior healthy rate.
2026-10-19	withdrawn	Withdrew this topic’s earlier six-signal/four-field claim and its receipt-only canary result/state exporters when integrating protected #2253. A fresh last-success receipt cannot establish the latest scheduled attempt or complete canary state. Reused the stronger indexed age producer instead of maintaining a competing implementation.	Earlier local adcbf54cc6/c7a7a8bdfa receipt tests established receipt validity only; protected main 3917df4297 supplies the indexed, byte-bound, hash-checked age source. No independently owned latest-attempt window exists.	Current source coverage is five of 15 signals and three of eight fields. Latest-attempt result/state, ten signal producers and five fields remain open; fixture passes are not operational qualification.
2026-10-19	implemented	Aligned the ephemeral CI database’s host port and service DSNs with canonical isolated validation port 5433 instead of relaxing the new source-regression guards. Added static CI-contract coverage for both accepted 5433 and rejected runtime-port 5432 configurations. Container-internal PostgreSQL still uses 5432.	Current change; CI 39196349170 attempt 1, exact head 50bd2a1bffe10edcb29c0700445470ee8299f42d, completed integration job 117879915744 step 9 reported fourteen fixture-port errors; no database outage or flaky-test claim. check-ci-contracts.py and the 159-case CI/artifact/fast-scope selection pass locally; the actual source-backend tests retain their loopback-5433 checks.	Repaired-head CI remains authoritative. This changes test infrastructure only, not runtime databases, deployed health, H01 coverage or operational qualification.
2026-10-19	implemented	Connected canonical consumed canary audit evidence to result, last-success age, and the canary field, with one bounded read-only latest-row query, complete-history fallback, exact receipt validation, preserved result freshness, and explicit source failure. Coverage is now six of 15 signal bindings and four of eight field bindings, not H01 completion.	current change; delivery/self_health_canary.py; delivery/self_health_sources.py; tests/delivery/test_self_health_canary.py; focused pytest with source/runtime/monitor regressions: 65 passed and 100% branch-inclusive canary-source coverage; Ruff and strict mypy passed.	Nine catalog producers and four complete field projections remain missing as listed in Remaining source boundaries. Scheduled-attempt completeness, primary/fallback delivery, and recovery-drill receipts remain open; no live or database integration validation was run.
2026-10-19	implemented	Added actual PostgreSQL regression evidence for the latest-canary reader without changing runtime bindings. Latest sequence, invalid latest evidence, empty history, and least-data bounded reads pass in isolated test schemas; the driver confirms read-only mode and the configured statement timeout.	current change; tests/persistence/test_self_health_canary_postgres.py; the single-file pytest selector through scripts/automation/with-local-test-database.py: four passed on loopback validation port 5433; Ruff lint and format passed.	Signal coverage remains six of 15 and field coverage four of eight. The prior row’s no-database-integration limitation is superseded only for this local query regression, not for deployment, notification delivery, or operational drills.
2026-10-19	implemented	Hardened shared scheduled-task interval validation before attaching daily stabilization jobs. Invalid NaN/Infinity and oversized conversion no longer enter schedule state or escape as unhandled compatibility-tool failures.	current change; #2254; 14 regressions failed before repair; 110 focused cases, actual creation command/reviewed-blueprint materialization, exact 60-second boundary, no-mutation updates, healthy due-cycle and RBAC/isolation controls passed; model/tool coverage 92.62%/97.63%.	Daily health/drift/deployment-baseline registration, recurrent-query signal, operating policy and operational receipts remain open; no authority or mode is raised.
2026-10-19	implemented	Bound the existing canary-age metric to the latest committed successful Core audit through a byte-bounded indexed PostgreSQL reader, without publisher/startup or oldest-prefix substitution. Reproduced and removed caller-precision rounding that hid a one-microsecond threshold breach.	current change; #2250; 99 focused checks including four actual isolated PostgreSQL cases; source and query branch-inclusive coverage 100%; natural partial-index plan, exact Decimal and byte boundaries, real Core producer and default readiness composition, concurrent index SQL and official Core migration validation.	Five signal slots/three fields are source-bound; ten/five remain unavailable. Apply the service-owned index before operational qualification. Latest-attempt result/state, complete audit history, delivery/drill and live receipts remain open.
2026-10-19	withdrawn	Withdrew e58fbf5674’s single embedding observation mapping to global adapter_healthcheck and its five-slot coverage claim. A configured embedding success plus an unobserved reasoner reproduced false aggregate Healthy.	Current correction; failing-then-passing collector regression; actual SDK producer, complete startup role registration, and strict Operator scope-consumer checks.	Global source coverage is four slots/three fields with 11/five unavailable. Model sub-observations are separately typed; complete adapter and replica coverage remain unimplemented.
2026-10-19	implemented	Connected a producer of actual normal embedding SDK outcomes to the registered-role and binding-qualified per-adapter slot. Overlapping calls must all settle; cancellation and incomplete outcomes cannot be overwritten by another success.	current change; embedding producer; provider_observations.py; focused actual MockTransport-call-to-existing-snapshot/reader-contract tests.	Five signal slots and three fields are source-bound; the adapter is t1.embedding only, while other roles and replicas remain unknown. Ten signals and five fields are unavailable. No rate, semantic quality, overall readiness, or deployment evidence is claimed.
2026-10-19	implemented	Bound actual runtime entry to first post-composition guarded-ready duration with monotonic elapsed time, frozen observation time, opaque process-instance identity, unavailable initial/shutdown states, and no additional probes.	current change; runtime/startup_observation.py; bootstrap/resources/readiness/source composition; focused startup, collection, shutdown, and bootstrap tests.	Four signals and three fields are source-bound; 11 signals and five fields remain unavailable. Model-health cooldowns and rolling latency samples lack outcome-rate denominators and cannot establish full adapter health. Fleet cold-start rate and deployed alert/drill evidence remain open.
2026-10-17	implemented	Added an exact exporter registry for all 15 self-health signals and eight unified operational-state fields. Runtime persists one source-qualified, freshness-checked snapshot, and the Operator exposes it only through the existing authenticated Reader-role projection boundary. Production uses explicit unavailable bindings until each accountable source adapter is composed, so missing telemetry never appears healthy.	current change; service contracts, Core monitor, production bootstrap, Operator projection and route; 171 focused checks and strict mypy.	Replace every explicit unavailable binding with its source-specific production exporter, then retain primary/fallback delivery and recovery-drill receipts.
2026-10-15	implemented	Added bounded read-only post-deploy verification for the exact consumed canary audit and its numeric latency budget. Publisher completion alone can no longer satisfy this smoke stage.	current change; verify_canary_audit.py; deploy workflow binding; 10 focused tests.	Retain a passing deployed receipt and add frozen fixture replay, kill-switch engage/disengage, and no-execute human approval dry-run evidence.
2026-10-15	implemented	Added the complete 15-signal self-health policy catalog, fail-closed freshness and threshold evaluation, notification-route fallback validation, and mandatory production startup binding.	current change; self-health catalog, decision core, readiness/bootstrap binding, and focused tests.	Bind every metric exporter and retain primary/fallback delivery plus recovery-drill evidence.
2026-08-14	in-progress	Adopted the implementation ledger and separated tested self-observability primitives from incomplete operational evidence; earlier provenance wasn’t reconstructed.	Current change; focused readiness, telemetry, canary, latency, and baseline-runner tests cited in the scope table.	Complete deployment round-trip evidence, alert and drill coverage, unified exposure, and stabilization bindings.
2026-08-14	implemented	Defined the operator-runbook front-matter schema and added CI coverage for every shipped ActionType.	Current change; check-action-runbooks.py and test_check_action_runbooks.py.	Keep deployment-owned commands and procedures aligned with the same required semantic sections.
2026-10-18	implemented	Bound actual local health-listener, guarded readiness, persisted kill-switch, installed version, and consumed policy sources. Added serialized 30-second source refresh independent of slow startup probes, sanitized source failure reasons, and cancellation cleanup without execution authority.	current change; delivery/self_health_sources.py; runtime/health.py; runtime/readiness.py; runtime/bootstrap_core.py; tests/delivery/test_self_health_sources.py; tests/runtime/test_self_health_sources.py; 92 focused pytest checks passed; Ruff lint/format passed; strict mypy passed for six changed source files.	Twelve signal sources and five operational-state sources still require their owning observations or projections. Primary/fallback notification delivery and live recovery-drill receipts remain open.

Remaining work

• Implement #2251’s bounded latest-result source with the pinned AKS schedule/controller witness and complete canonical sequence; 153 focused checks cover every unavailable/nonpassing case and prevent earlier-success reuse. Unsupported or unconfigured witnesses stay unavailable.
• Bind and qualify additional native scheduler witnesses, including Container Apps, without treating an execution name or publisher acceptance as proof of the latest due occurrence.
• Require the exact consumed canary audit entry within a 30-second numeric latency budget after publisher completion, rejecting ambiguous, malformed, unsettled, future-dated, or over-budget rows.
• Retain a passing deployed canary receipt, replay the frozen observation mode fixtures, exercise kill-switch on/off, and complete a no-execute human-approval dry-run with linked audit receipts.
• Map every self-health signal to a concrete threshold, generic owner lane, notification route, and fail-closed fallback, and validate the complete catalog during production startup.
• Bind every catalog signal to its runtime exporter and retain primary/fallback delivery and recovery-drill receipts in one governed deployment.
• Bind process-local liveness, guarded readiness, and persisted kill-switch state plus installed version and consumed runtime policy, and test periodic refresh without repeating startup probes.
• Observe actual runtime entry to first guarded-ready duration per process without inventing a cold-start rate, renewing freshness, or performing another model, network, or database probe.
• Expose the complete version, configuration, rule-effect, override, discovery, canary, kill-switch, and break-glass state through one authorized read surface and test field freshness.
• Define a required runbook schema and add a check proving every automated ActionType has a generic upstream or deployment-owned runbook with verification and rollback instructions.
• Register the metadata-only recurrent-query input through the actual typed plan and Operator/Bragi/Norns/Mimir path (#2271 / PR #2273; local mechanics only).
• Register the daily stabilization jobs, then prove the window opens, lowers authority on guard-metric breach, and closes only after its configured exit conditions.
• Complete the live Azure guest-event, provider-throttling, and cross-service parity evidence tracked by the Azure read-investigation owner document.

Startup readiness response

Use /live to determine whether the process can answer, and use /ready to determine whether it may process events. A 503 response means a process-critical startup probe is missing, stale, timed out, crashed, or failed. Don’t restart consumers or the Pantheon manually while /ready remains closed.

1. Read the sanitized latest report at the StateStore key runtime:startup-readiness:latest. Start with decision, missing_probe_ids, stale_probe_ids, and each result’s failure_class.
2. Correlate a decision change with the startup_readiness.transition audit record and the schema-validated readiness_transition event. A publish failure has its own startup_readiness.transition_publish_failed audit record.
3. Repair the named dependency or capability outside the FDAI process. Avoid placing provider error text, endpoint values, tokens, or customer identifiers into the report or an operator note.
4. Wait for the configured periodic refresh. A recovered process-critical probe reopens /ready and restarts guarded workers. A recovered capability does not raise authority above its deployment promotion state.

For degraded, keep /ready open but inspect authority_ceilings. shadow, human_approval, deterministic_fallback, and disabled are expected safety responses, not permission to bypass the quality gate or promotion registry. See startup-and-lifecycle.md for probe budgets and the registered-destination contract.

Signals emit via OpenTelemetry to the configured backend (deployment.md#observability-slos-and-alerting).

OTEL_EXPORTER_OTLP_ENDPOINT enables OTLP/gRPC trace and metric export. HTTPS is required outside loopback; credentials, query strings, and fragments are rejected in the endpoint. Without an endpoint, local console spans and in-memory metrics remain the default.

Channel, extension, model, scheduler, and security lifecycle components emit the same fdai.transition span and fdai.transition.count metric through a process-singleton emitter. Attributes use bounded allowlisted domain, name, outcome, and component-specific scalar keys; provider error text, payloads, credentials, and arbitrary labels are not accepted. Emission is best-effort so exporter failure cannot block routing or safety decisions.

Synthetic Canary Event

A mostly event-driven system has a specific silent failure mode: no events arrive -> looks healthy. Mitigation: a periodic canary on a separately authorized topic.

• A synthetic event is emitted every five minutes from a Container Apps Job into fdai.control.canary on the same Event Hubs namespace.
• A dedicated UAMI can only pull the image and send to Event Hubs. The core’s separate canary consumer accepts only source=fdai.canary-job and event_type=fdai.control.canary.
• The canary path records ingest, route, and audit stages plus a no-op audit entry. It never enters T0/T1/T2, the safety check, execution, IRP, or the learning loop.
• The full loop - ingest → correlation → tier decision → audit entry - MUST complete within a bounded budget; a failure to complete raises an SLO-burn alert on the operational alert-routing lane.
• The canary is versioned, rate-capped, and its idempotency key is distinguishable from a real event’s so canary samples cannot corrupt regression measurement or the autonomous discovery loop’s observe stage.
• Each five-minute slot derives a stable UUID and canary:<slot> idempotency key. Container Apps caps publisher execution at 120 seconds and the audit row records measured latency.

The deployment workflow now blocks when an immediate canary publisher run fails or its exact consumed Core audit row is absent, ambiguous, malformed, unsettled, future-dated, or over the 30-second latency budget. A retained live receipt and scheduled kill-switch on/off drill remain production-readiness evidence items.

Post-Deploy Smoke Test Contract

The target automated suite runs against the live deployment after every promotion. A failing smoke test should abort the promotion and roll traffic back (deployment.md#release-and-rollback).

1. Adapter reachability - Kafka round-trip (Event Hubs :9093 produce + consume on a probe topic), Key Vault reference resolution, DB write + delete on a probe table, T2 model endpoint low-cost ping (per model, including cross-check target).
2. Config load - the deployed image reports its version, catalog ref, and config hash; values match the expected release manifest.
3. Canary round-trip - fire one synthetic event, verify the audit entry lands within budget.
4. Observation mode decision correctness - a fixture set of representative events is fed in observation mode mode; decisions match golden expectations (regression suite).
5. Kill-switch check - toggle kill-switch on, verify all actions hold for review during the window (probing with the canary); toggle off, verify normal decisions resume. Both states leave audit entries.
6. human approval dry-run - a synthetic high-risk detected issue is routed to the human approval channel, an approver approves (in a dry-run harness that does not execute), the audit trail records both hops.

The current apply workflow verifies schema migrations, optional HTTP health endpoints, publisher completion, and the exact consumed canary audit within a 30-second budget. A retained live receipt, fixture replay, kill-switch drill, and human approval dry-run remain required before production promotion.

Alert Routing

Two independent lanes, each with an owner and a channel. Concrete channel names / ownership matrix is fork responsibility. Channel selection, trust tiering, and fallback rules are defined in channels-and-notifications.md; this section is the alert-side view of that model.

Lane	Signal source	Route
Operational	SLO burn, DLQ depth, verifier failure rate, cold-start deadline miss, adapter unhealthy, canary miss, IaC drift, secret near expiry	on-call rotation (paging channel)
human approval	high-risk detected issue, enforce-promotion request, override request, exemption-expiry warning, break-glass request	Teams human approval channel

Rules that apply to every alert:

• Alerts MUST be actionable: each alert links to (a) its dashboard panel, (b) its runbook, (c) the correlated audit id if applicable.
• De-duplication: correlated alerts collapse per the correlation rules in observability-and-detection.md; an alert storm from one root cause is one page, not many.
• Fallback channel: if the primary channel (Teams / paging) is unreachable, human approval items queue in the state store and alert via a secondary channel; nothing auto-executes on the fallback path.

Open decision: Choose the concrete channel-ownership matrix and fallback channel per fork.

Audit Investigation Flow

Given a correlation id or audit id, the operator walks a fixed path. Each hop is a stored link captured at write time, not a search - the walk is O(1) lookups.

The audit record is append-only and hash-chained per security-and-identity.md; the same walk works for observation mode and enforce events (mode is recorded on every entry).

Runbook Set

Every automated action should have an operator-facing runbook. Upstream ships generic operational runbooks under docs/runbooks/; deployment-specific values and procedures stay in a fork-local runbook set per generic-scope.instructions.md. The fdai_runbook front-matter block declares the schema version, covered ActionType patterns, and the headings that provide preconditions, procedure, verification, rollback, and audit guidance. check-action-runbooks.py fails CI when a shipped ActionType has no runbook, matches more than one runbook, or a declared required heading is absent.

Runbook	Purpose	Trigger
Kill-switch drill	halt all auto-execution, verify all paths hold for review	operational incident, scheduled drill
DLQ drain	inspect, replay, or discard dead-lettered events (with idempotency-key guards)	DLQ depth alert
Drift reconciliation	reconcile IaC drift via a PR (never silent apply)	scheduled drift alert
Application rollback	shift traffic back to the previous container revision	SLO burn, error spike, smoke-test fail
Action rollback	revert a per-action change (git revert, snapshot restore, replica-promotion undo)	rollback request, auto-demotion
DR failover	fail the control plane to an alternate region from state + backups	region outage
Override withdrawal	remove an active override, re-enable the underlying rule on that scope	rule revised, risk changed
Catalog rollback	revert to the previous rule-catalog version	bad rule set promoted
Break-glass	grant scoped emergency access under audit + auto-expiry	verified emergency

Every runbook MUST state:

• Preconditions (permissions, prerequisite alerts).
• Exact commands (or the exact console navigation), copy-pasteable.
• Verification (what to check that proves it worked).
• Rollback of the runbook itself (undo of the operator step).
• The audit trail the runbook leaves.

Deployment boundary: The upstream schema and generic procedure are enforced in CI. A fork supplies provider commands, exact resource values, and any narrower deployment-owned runbooks.

Version and Configuration Exposure

The system MUST expose, at any time, machine- and human-readable, without special access:

• Deployed image digest and semantic version tag.
• Rule catalog version tag + content hash.
• Config hash (a stable sum over live runtime configuration; secrets excluded).
• Per-rule effect + enforcement flag - “what is currently enforced” for each rule / scope.
• Per-scope override count (linked to a list view).
• Autonomous discovery loop state (enabled / disabled, last cycle timestamp, last cycle pass rate).
• Time since last successful canary round-trip.
• Kill-switch state and break-glass usage in the current window.

Content only; presentation / dashboard layout is defined separately.

Pre-Launch Verification (performance + integration)

Before a service opens, FDAI is most useful run in observation mode alongside a performance / integration test of the workload it will watch. FDAI does not generate the load - an external load generator (Azure Load Testing, k6, JMeter) drives the traffic - but while that traffic runs, the control plane proves its detection and judgment against realistic conditions without acting:

• Observation mode judgment under real load. New rules and actions run judge-and-log only (architecture.instructions.md § Observation mode -> Enforce), so the load test exercises the deterministic tiers and the T2 quality gate and every decision is recorded, none executed.
• Detection latency measured against budget. The events the load generates feed the per-tier LatencyBudgetMonitor (core/measurement/latency_budget.py), so a tier that misses its p95 budget under load surfaces before go-live, not after.
• Canary + smoke round-trip. The synthetic canary and the post-deploy smoke test contract confirm the full ingest -> tier -> gate -> audit loop completes within budget on the loaded environment.
• Scenario replay. tools/baseline_run.py replays the frozen scenario set (goals-and-metrics.md) so routing and auto-vs-human approval accuracy are quantified on the same build that will ship.

The result is a body of observation mode evidence - accuracy, latency, zero policy-violation escapes - that an operator reviews before promoting any action from observation mode to enforcement mode.

Azure read-investigation release evidence

Live checks use existing resources and a reader credential. They do not create, update, start, stop, or delete an Azure resource. Repository tests use synthetic, customer-neutral payloads for failure paths that are not safe to induce against a live subscription.

Scenario	Evidence class	Result
Successful caller attribution	Live	Passed. Exact resolution and projected Activity Log reads matched user and service-principal actors while retaining only opaque actor and correlation references.
Resource Health	Live	Passed. An empty ARG projection fell back to the current Resource Health REST endpoint and returned normalized availability evidence.
Unauthorized scope	Live	Passed. An inaccessible scope became unavailable with a failed bounded receipt.
Ambiguous resource name	Live	Passed. One duplicate name returned four bounded candidates, no exact resource binding, and no history query.
Guest OS shutdown	Live and contract	Incomplete. Sixteen accessible workspaces contained no retained Event or Syslog shutdown record across their available history. Live missing-workspace behavior returned unavailable; matched Event and Syslog normalization passed contract tests only.
Provider throttling	Contract	Behavior passed. Synthetic 429 responses exercised bounded retry and terminal failure. An actual live 429 was not induced because deliberate throttling would violate the bounded-read policy.
Insufficient retention	Contract	Passed. Lookbacks beyond configured Activity Log or guest-log retention fail before HTTP and normalize as unavailable through the provider boundary.

The incomplete guest-event row and missing naturally occurring live 429 remain release evidence, not implementation defects. Keep the issue open until the dedicated validation environment can produce those observations without an Azure change.

Post-Launch Stabilization Window

After a service opens, FDAI is most useful left running for the first few days at a heightened observation intensity - the stabilization window. It is the leading edge of the 30-day measurement window, not a separate mode, and it composes existing primitives:

Implementation status: Scheduler/measurement primitives and the metadata-only console.recurrent_query input are implemented. Daily health/drift/deployment-baseline jobs, installed transport/cohort qualification and full stabilization lifecycle evidence remain open. The bullets below define the target stabilization composition, not a completed rollout.

• Observation-first stays the default. Newly introduced actions remain in observation mode through the window; promotion to enforcement mode waits until the stabilization signals below are clean, so an unstable opening never auto-executes.
• Scheduled comparison to baseline. Scheduled tasks (core/scheduler) run daily health checks, configuration-drift diffs, and deployment verification against a documented baseline (including an uploaded resource plan in the knowledge base) - exactly the “compare to baseline” checks the operator wants right after launch.
• Pattern promotion from real traffic. The Month-1 observation tools and the console.recurrent_query signal (operator-console.md § 9.3) turn a repeated investigation into a rule candidate, so the catalog grows from what the launch actually surfaced.
• Close guard-metric watch. Guard-metric drift (goals-and-metrics.md § Guard Metrics) is watched tightly through the window; a breach demotes back to observation mode automatically. When the signals settle, operation returns to the normal cadence.

The window absorbs the noisy opening period with minimal human intervention, then hands off to steady-state operation once stabilization signals hold.

Open Decisions

• Synthetic canary audit-freshness and numeric round-trip alert budget. The publisher cadence defaults to five minutes and the canonical payload/idempotency shape is implemented.
• Smoke-test suite composition (fixture set, per-step budgets, promotion-gate wiring).
• Alert channel ownership matrix (fork vs upstream) and the fallback channel selection.
• Runbook template - required sections, format, and CI check that a runbook is present for every automated action.
• Retention window and query model for the audit investigation flow.
• Cold-start deadline value (shared with startup-and-lifecycle.md).
• Stabilization-window length after launch (default “a few days”) and the concrete stabilization signals that end it (guard-metric quiescence, canary streak, scenario-replay pass).
• Pre-launch load-test integration surface (which load generator, what per-tier latency budgets to assert under load).