Channels and Notifications
Core supervises post-turn review and document-intent ready intake separately from channel, notification and approval topics; both use independent event-bus groups and add no human channel, notification recipient, approval transport or authority. Its read-only ActionBuilder rule map validates ORR citations and actual verifier/native comparison-window observations feed self-health; neither creates a channel, delivery receipt or approval route, or selects the notifications add-on.
How FDAI talks to humans through Teams, Slack, email, webhooks, paging services, SMS, a Direct Line-compatible custom channel, and opt-in browser notifications. This file is authoritative for the channel abstraction, trust levels, category boundaries, routing policy, and channel-specific rules. It resolves the placeholder “notifier interface” hinted at in tech-stack.md and consolidates the Alert Routing fragments from alert-routing contract and the Teams-specific flows in user-rbac-and-identity.md.
The read-only console’s identity and interaction flows remain out of scope; this document owns only its outbound browser-notification boundary. Console identity lives in user-rbac-and-identity.md.
Product profile: The observation-first default constructs no channel. Notifications require the explicit
notificationsadd-on; A1 approval additionally requires governed execution and complete enterprise identity evidence. Environment values alone cannot select either add-on.
Production A3 ingress, publishing, lifecycle, persistence composition, deployment, and rollback are owned by Production A3 channel runtime.
Core supervises Assurance Twin’s read-only report/review outbox separately from the A1 request outbox. A Twin evidence gap is not an approval decision or a notification-channel fallback; neither worker can drain or authorize the other’s records.
A2/A4 fan-out delivery to every enabled and configured channel, named channel bindings, per-channel durable delivery, and the Teams Workflows webhook binding are owned by Multi-channel notification delivery.
Direction scope. Outbound notifications, A1 approvals, and bidirectional conversations use separate Protocols. This document owns their shared trust/category/routing principles and outbound delivery; operator-console.md owns conversation tool and session semantics. Adapters can share credentials, but the three contracts keep distinct blast radii.
Customer-agnostic: every channel id, group name, and endpoint below is a placeholder. A fork supplies its own tenant, workspace, and endpoint values via config (generic-scope.instructions.md).
1. Design Principles
Section titled “1. Design Principles”- Three narrow abstractions, many adapters.
NotificationChannelowns A2/A4 push,HilChannelowns A1 send/poll, andConversationChannelAdapterowns A3 inbound/outbound. Core code never names Teams or Slack; new vendor adapters are additive. - Categorize by purpose, not vendor. A channel supports one or more of four categories (§3). Vendors are constrained by which categories they can safely serve.
- Trust-tiered. Approval-category traffic (A1) MUST NOT flow through a channel that cannot verify the human’s Entra identity end-to-end. A less-trusted channel MAY carry information but never decisions (§4).
- Choose the safer default when the outcome is uncertain. If every configured channel for a category fails, the request queues and pages the operational lane - it never auto-executes. Fallback within a category preserves the trust tier (§6).
- Redaction is the sender’s job. No secret, credential, PII, subscription id, or raw customer payload leaves the trust boundary in a channel message. This applies to every category, not just approvals.
2. Where Channels Sit in the Architecture
Section titled “2. Where Channels Sit in the Architecture”
- Outbound adapters live under
delivery/notifications/, bidirectional adapters underdelivery/channels/, and A1 approval adapters underdelivery/chatops/. Their contracts live inshared/providers/notifications/,conversation_channel.py, andhil_channel.py, respectively. - The channel-router is a thin core module: it takes a category and a message and picks channels per the fork’s routing config (§6). It holds no vendor knowledge.
- Approval callbacks from any adapter land at
fdai-api, which re-validates the human’s Entra identity (user-rbac-and-identity.md) before acting. Adapters never authorize decisions themselves. A channel approval never becomes a full-authority development confirmation. Only the FDAI Console decision route can attach one, as a token-free fresh-sign-in attestation that the decision requiring human approval payload carries to Core for revalidation; Slack and Teams decisions never carry it. The subscription tag pilot keeps this Console-only boundary whether the isolated Executor uses a gateway or its direct Azure adapter. - A1 approval parks may carry the active single-operator production approval-profile metadata: profile revision, policy digest, original quorum, effective quorum, operator principal, and executor principal. Core pins the digest-verified revision at decision start from the active Mimir approval-policy pointer, or from the bootstrap environment fallback before activation. Channel adapters treat these fields as immutable context for the callback and audit; the Operator API and Core revalidate them before any approval is accepted. The metadata does not let Slack, Teams, or a notification route select a profile or grant execution authority, and a later activation does not rewrite an in-flight park.
- The
governance.override-promote-action-typeapproval uses the same A1 path. The channel carries the pinned profile and exact approval receipt context, while Thor alone can write the promotion registry after Var approval is verified. - A1 approval parks may also carry a pinned operator admission-policy digest. Channel adapters display or return it only as immutable approval context; they never re-read the policy pointer or evaluate Rego.
- Core policy-revision signature verification happens before the channel layer sees approval context. It changes no channel trust tier, callback route, fallback behavior, or notification authority.
- Core’s runtime task supervision runs the license entitlement-state publisher as its own background task beside the channel outbox tasks. It sends no channel message, and a slow or failed publication cannot delay Slack, Teams, or notification delivery.
3. Categories (A1-A4)
Section titled “3. Categories (A1-A4)”Every channel message carries a category tag and must obey that category’s rules.
These are notification.a1 through notification.a4 machine categories. They are not the
Constitution’s autonomy.a0 through autonomy.a4 authority classes, and no numeric suffix implies
equivalence or conversion between the enum families.
| Category | Direction | Examples | Auth strength needed |
|---|---|---|---|
| A1 - human approval approval | bidirectional (decision returned) | high-risk action approval, enforce-promotion approval, exemption approval, override approval | highest - verified Entra identity, action-bound, no replay |
| A2 - Operational alert | outbound only | SLO burn, DLQ depth, verifier failure rate, cold-start miss, IaC drift, adapter unhealthy, canary miss, urgent or incomplete code-security scan | low - informational |
| A3 - Chat command | bidirectional (query/response) | read: /aw status, /aw shadow-report, /aw override list, /aw kill-switch status. write (draft-PR only): /aw override draft, /aw exemption draft, /aw assignment param-tune | medium - role-gated per command (see §3.1) |
| A4 - Digest | outbound only | daily observation mode-accuracy report, weekly override retrospective, weekly enforce-promotion candidates, weekly governance PR aging, weekly exemption expiry lookahead, monthly KPI + cost roundup, break-glass usage summary, daily code-security digest | low - recipient scope only |
Category boundaries (MUST)
- A1 approvals carry opaque bindings, never identity, authority, or a decision payload. The receiver derives the approver from the authenticated transport and delegated token, then revalidates the exact card contract. See Operator approval callback wire contract.
- A3 write commands never mutate the live catalog directly - they produce a draft PR the same way the console does (§6 in user-rbac-and-identity.md), carrying the invoker’s Entra OID in the PR trailer. The PR then follows the standard quorum + no-self-approval rules.
- A2/A4 messages never contain approval buttons or executable links. The scheduled exemption lifecycle sweep sends its ahead-of-expiry digest as one A2 alert through this router, with an audit id derived from the exact exemption revisions so a repeat is suppressed (Rule Governance).
- Learned patterns and predictions create no A1 request in the default profile. Without the governed execution add-on, Forseti answers forecasts and capacity-fed arbitration with ActionType-free Decisions, and the control loop stops T1 learned reuse before it builds an Action, so no channel carries an approval for them. See the learned and predicted output boundary.
- A T2 rubric decision creates no A1 request. The rubric leg only lowers confidence or holds a T2 candidate, even when a verified receipt enforces it for an ActionType, and T2 candidates do not become Actions until the gated downstream bridge exists. See the Hallucination Rubric Gate.
- Causal hypotheses create no A1 request. A projected or closed
CausalHypothesisrevision is evidence for Forseti’s judgment only. Neither the runtime’s Forseti projection nor its Thor ActionRun receipt resolver sends a channel message or opens an approval request.
3.1 A3 command role gating
Section titled “3.1 A3 command role gating”Each A3 command declares a minimum role and whether it is read or write. The bot
adapter enforces the check via the invoker’s Entra OID (Teams SSO / Slack mapping) before
running the handler; a missing role responds with 403 in-channel and writes an audit
entry.
| Command | Type | Min role |
|---|---|---|
/aw status, /aw shadow-report, /aw kpi | read | Reader |
/aw override list, /aw exemption list, /aw kill-switch status | read | Reader |
/aw override draft, /aw exemption draft, /aw assignment param-tune | write → draft PR | Contributor |
/aw kill-switch on/off | write → draft PR + A1 approval | Owner |
4. Trust Levels (matrix)
Section titled “4. Trust Levels (matrix)”A channel’s allowed categories are the intersection of what it can technically deliver and what its authentication can prove.
| Channel | Entra tenant | Auth path | Categories allowed |
|---|---|---|---|
| Teams (same tenant) | ✓ | Teams SSO → OBO exchange → fdai-api token | A1, A2, A3, A4 |
| Teams (guest tenant) | guest | OBO with guest OID | A2, A3, A4 (A1 denied - same guest rule as user-rbac-and-identity.md §10.5) |
| Slack | ✗ | Slack OAuth; fork-mandatory Slack userId ↔ Entra OID mapping; signed interaction and browser Entra reauthentication for A1 | A1 only with a non-empty mapping and fresh signed API-token auth_time; A2, A3, A4; outbound and handoff are locally implemented, not live-validated |
| Email (SMTP / Graph) | ✗ | send-only, no return channel | A2, A4 only - never A1 (magic-link approvals aren’t supported) |
| Generic webhook | ✗ | HMAC-signed, timestamped, replay-guarded | A2 only |
| PagerDuty / Opsgenie | ✗ | API key, ack from mobile app | A2 only (operational lane paging) |
| SMS | ✗ | - | A2 only (minimal payload; break-glass reachability) |
| Direct Line-compatible custom channel | ✗ | Relay secret from protected configuration; posts activities only | A2, A4 only - never A1 or A3 |
Rules that keep the matrix safe (MUST)
- Magic-link approvals aren’t supported across every channel. Approval always requires
a re-authenticated round-trip through
fdai-api. - A1 fallback stays inside A1-capable channels. A failed Teams A1 attempt never falls through to email; it falls to another A1-capable channel (Teams standby, or Slack only after mapping, browser actor binding, and durable send reconciliation) or to the human approval queue.
- Slack A1 decisions require the userId↔OID mapping. The outbound buttons carry only the parked approval id, not an actor, role, action hash, or approval decision. The separate signed callback refuses to authorize a response without a non-empty mapping for the responding Slack user; a missing mapping is “no approver” (fail-closed to the human approval queue).
4.1 Sender pairing trust bootstrap
Section titled “4.1 Sender pairing trust bootstrap”ChannelAccessService supports disabled, allowlist, and pairing per channel. The durable
PostgreSQL store serializes request creation with a channel-scoped transaction lock, enforces the
pending cap atomically, excludes expired requests from that cap, and conditionally approves the
stored digest. An approved sender cannot re-enroll and overwrite its principal mapping.
NativePairingChallengeFlow sends the plaintext challenge only as a same-thread channel reply;
the store and response metadata retain only the SHA-256 digest and expiry. A delivery failure
conditionally deletes the matching pending digest so an undelivered code cannot consume capacity.
Approval still requires a distinct authorized actor and an existing FDAI principal. Pairing grants
identity resolution only; it never grants a role or bypasses the coordinator’s tool RBAC.
Cross-channel identity links are explicit relation records above pairing, not identity merges. Both senders MUST already be independently approved for the same FDAI principal, the link MUST span two channel kinds, and a distinct authorized actor MUST approve it. If the sender mappings name two different principals, the service rejects the request before any write. The deterministic link id makes retries idempotent, and the PostgreSQL record survives restart without modifying either sender mapping or either principal’s role.
Channel attachments are evidence inputs, never instructions. Slack and Teams adapters can normalize
only bounded file metadata and an opaque vendor id; payload-supplied download URLs are discarded.
The current production A3 composition does not bind a private fetcher or protected ingestor. It keeps
attachment support off and returns 422 attachments_unavailable before queue admission. Setting
FDAI_CHANNEL_ATTACHMENTS_ENABLED=1 fails startup until the versioned document-ingestion handoff is bound. A direct queue injection also fails before claim or semantic publication. Future
fetch and ingestion must preserve the original text, admit only READY doc: citations, and retain
the malware, protection, indexing, access, retention, and agent-ownership gates.
Teams ingress separates two identities. TeamsServiceTokenVerifier verifies the Bot Framework
service token against cached JWKS with RS256 signature, app audience, Bot Framework issuer,
expiration/not-before, and required serviceurl. The route then requires the activity
serviceUrl and channelId=msteams to match that verified service identity. Only after that check
does ChannelPrincipalResolver validate the activity tenant and map from.aadObjectId to a bounded,
configured canonical FDAI principal. A service-token failure returns 401; an unknown tenant or
user binding returns 403; neither reaches the channel queue. The vendor id is replaced by the
canonical principal before the conversation gateway sees the turn.
Production composition reads FDAI_TEAMS_APPLICATION_ID, optional HTTPS issuer/JWKS overrides,
FDAI_TEAMS_TENANT_ID, and FDAI_TEAMS_PRINCIPAL_MAP_JSON. The binding map is a non-empty
string-to-string JSON object capped at 1000 entries. Missing, malformed, or unbounded configuration
fails at startup. The Bot service token authenticates the channel service; it never substitutes
for the operator’s Entra principal or grants an FDAI role.
ChannelEdgeRuntime is the implemented library runtime for a standalone Operator-distribution edge
process. It isn’t mounted into the Operator API process and never receives the executor identity.
The production ASGI factory and Terraform workload register only enabled bounded ingress routes and
start one Operator semantic-turn consumer per adapter. Missing credentials, Teams identity,
endpoint policy, JWT configuration, principal scopes, or persistence fail startup before a route
accepts traffic. Attachment enablement without a production ingestor fails at the same boundary.
Shutdown closes queues, consumers, provider clients, and credentials exactly once.
Channel enablement uses FDAI_CHANNEL_EDGE_ENABLED_CHANNELS; queue and request bounds remain
server-owned. Container Apps native Key Vault references populate the Slack and Teams secret
environment values without placing secret values in source or Terraform variables. /health/live
and /health/ready expose only content-free process state.
4.2 Rich thread and delivery behavior
Section titled “4.2 Rich thread and delivery behavior”OutboundResponse carries vendor-neutral rich delivery and thread intent without giving core code
a Slack or Teams dependency. Existing text replies remain the default; scheduled continuations use
explicit origin or dedicated thread mode and carry an opaque anchor id as metadata. A response can
add bounded mentions and one rich operation; ambiguous or oversized values fail before publishing.
Read observation replies can also carry an ordered, bounded activity sequence. A handoff names
Bragi as the conversational router and the accountable observer, while observed execution records
the observer, canonical server tool, redacted command, safe result summary, status, authority, and
timing. The complete sequence is part of durable response replay and never grants channel adapters
tool or execution authority. In addition to per-field limits, the complete activity sequence has a
48,000-character budget so one accepted response cannot exceed durable or vendor payload ceilings.
Command, output, label, tool, and authority fields all pass the same high-signal secret scanner;
Bearer credentials are rejected across whitespace and common separator forms.
Durable activity decoding is type-strict and validates ordered RFC 3339 timestamps before a
publisher receives the response.
Concrete publishers map that intent as follows:
Publisher transport and acknowledgement handling stay separate from pure bounded Slack Block Kit and Teams Adaptive Card rendering. This split preserves wire payload and fallback behavior and keeps vendor presentation independently testable.
Presentation capability rendering
Section titled “Presentation capability rendering”One validated ChannelPresentationEnvelope carries canonical text, a versioned presentation
artifact, exact evidence references, limitations, authority, and an optional server-owned Web link.
The envelope is the only input to a ChannelPresentationRenderer. A renderer returns a bounded
vendor payload and the complete readable text fallback; it cannot fetch evidence, run a tool,
authorize a request, or change a fact.
ChannelPresentationCapabilities owns provider limits and support for text bytes, serialized
bytes, block count, fields, actions, images, deterministic sparklines, edits, threads, and progress.
The core planner never branches on a vendor. A deployment injects a renderer and capability
profile, and an unknown or failed renderer returns the canonical text fallback.
| Surface | Rich rendering | Chart degradation | Mandatory retained content |
|---|---|---|---|
| Teams | Adaptive Card FactSet, Container, and ColumnSet | Textual trend plus top facts and a Web action unless a verified image renderer is injected | Limitations, evidence references, authority, unavailable state, and canonical text fallback |
| Slack | Block Kit section, fields, context, and actions | Bounded textual summary; a sparkline is allowed only from a reviewed deterministic renderer | Limitations, evidence references, authority, unavailable state, and top-level text fallback |
| Custom | Injected renderer and capability profile | Renderer-declared bounded fallback | The same mandatory content and bounds as built-in renderers |
Conversation assurance evaluates this projection through an independent capability profile. Canonical content, limitations, evidence references, and authority posture are always applicable. Preparing status, progress, activity records, rich presentation, thread continuity, and edit continuity apply only when the injected profile declares support. Direct Line and custom adapters use this same profile seam. This A3 presentation contract doesn’t implement a Direct Line conversation transport or add a vendor branch to Core; A2/A4 Direct Line-compatible notification delivery is owned by Multi-channel notification delivery § 9.
Rendering is loss-bounded, not fact-selective. It can omit optional visual detail only after the readable fallback retains the same canonical facts. It never maps unavailable to zero, drops an approval or authority boundary, or emits raw artifact JSON as the primary answer. Provider byte, block, and field caps apply before transport. Exceeding a cap first removes optional visual detail, then uses the complete bounded text fallback. If mandatory content alone cannot fit, rendering fails closed before a provider call.
The pure payload builders and fake custom renderer remain independently testable. The standalone Operator edge binds the Slack and Teams builders to authenticated ingress, fixed HTTP transports, strict acknowledgements, and durable replay. Governed provider and deployed runtime evidence remains open work.
| Behavior | Slack | Teams | Text fallback |
|---|---|---|---|
| Thread reply | chat.postMessage with thread_ts | Message activity with replyToId | Same originating thread |
| Mention | <@vendor-id> | <at> text plus a Bot Framework mention entity | @display-name; the opaque target id is omitted |
| Streaming | Initial chat.postMessage, then cumulative chat.update | Initial activity POST, then cumulative activity PUT | One final text reply |
| Edit | chat.update for the declared message id | Activity PUT for the declared activity id | New thread reply prefixed with Update: |
| Reaction | reactions.add against the inbound message | messageReaction activity against the inbound message | New thread reply with a Reaction: label |
| Agent activity | Block Kit sections in handoff, plain-text command/result, Bragi-answer order; posts, stream updates, and edits preserve the same blocks | Adaptive Card blocks in the same order under a 24,000-byte serialized card budget; multibyte answers are byte-bounded and omitted activities are counted | Bounded text with the same agent attribution and redaction markers |
Slack keeps the complete activity fallback in the top-level text field for notification and
accessibility fallback. Block Kit carries each activity once in its own section and passes only the
canonical answer to the final Bragi section, so structured clients do not render duplicate evidence.
Progressive conversation delivery uses typed cumulative ChannelProgressUpdate snapshots rather
than splitting a completed answer into artificial token chunks. The gateway derives snapshots only
from ordered coordinator activities and the canonical final preview. Revisions are contiguous,
activity counts never decrease, and the last snapshot is confirmed only when its text and activity
count exactly match the durable OutboundResponse. Slack and Teams post revision 0 once, then edit
the same acknowledged message for later revisions. A response without observed activities remains
a single final post. A channel with streaming disabled sends the same canonical final text fallback.
Running revisions contain the bounded progress summary only. The final confirmed revision adds the
canonical redacted activity blocks, so command and output evidence does not expand while work is in
progress but remains inspectable after completion.
Production composition can inject the shared progressive-conversation collector into both
publishers. It records aggregate first-progress/confirmed latency, truncation, terminal completion,
and post-acknowledgement ambiguity without retaining message, destination, or identity values.
Truncation includes output clipped to a vendor field limit and any complete activity omitted from
the Teams Adaptive Card to preserve its 24,000-byte budget. It also includes a canonical Teams
answer clipped to the 4,000-character answer-block limit and a canonical Slack answer clipped to
the 2,900-character Block Kit section limit after mrkdwn escaping. The metric records only that
truncation occurred. When multibyte JSON encoding reaches the Teams card byte ceiling first, the
answer is clipped earlier with the same marker and metric. The omission count and clipping marker
stay visible in the vendor message, and the original activities and answer remain in the durable
response evidence.
Observed output uses explicit provenance markers. [UPSTREAM OUTPUT TRUNCATED] means the evidence
producer supplied partial output; [CHANNEL OUTPUT TRUNCATED] means the adapter clipped output to
the vendor presentation limit. Both markers appear when both conditions apply.
The concrete Slack and Teams configurations own capability flags for mentions, streaming, edits, and reactions. A disabled capability never causes the core to guess a vendor payload; the publisher uses the documented text fallback instead. Thread context is preserved during every fallback. Vendor endpoints remain fixed in publisher configuration or the authenticated Teams endpoint resolver. Response data cannot supply a URL, token, or alternate API method.
Every accepted send returns a ChannelDeliveryReceipt with the requested operation, vendor message
id, and whether the request degraded to text. Slack requires an ok=true response and message
timestamp for posts. Teams requires the Bot Framework resource id for message creation. Missing or
malformed acknowledgements fail the send rather than reporting delivery. The adapters forward the
receipt to the caller; transport failures still raise and follow the existing retry/audit path.
After the initial post is acknowledged, any later update failure is classified as ambiguous
duplicate risk. Durable delivery does not retry the complete stream and create a second message.
4.3 Durable reply delivery and adapter controls
Section titled “4.3 Durable reply delivery and adapter controls”External conversation replies use the persisted ledger in durable-conversation-delivery.md. Provider HTTP rejection is a definitive failure eligible for bounded retry. Transport interruption or a missing/malformed acknowledgement is ambiguous and never retried automatically. Pause/resume mutations live only in separately authenticated ChatOps command routes; the console receives GET-only reliability metrics.
Authenticated generic webhooks can opt into TypedWebhookMapping. The mapping fixes one
allowlisted normalized event type and target agent at configuration time; payload-supplied event,
agent, command, or session values cannot override them. Server-owned dot paths project only
bounded scalar fields. Missing fields, containers, oversized strings, or non-allowlisted targets
fail before publication. A bounded session key is the SHA-256 digest of explicitly selected scalar
values, so raw external identity values do not become session ids. The projected event still
enters event-ingest, trust routing, risk gating, and audit; the webhook never executes an action.
4.4 Browser system notifications
Section titled “4.4 Browser system notifications”The explicit client-local console-web channel is owned by
Console Web Notifications. Settings is its canonical
principal-and-browser selection surface, the header is a synchronized shortcut, and only
runtime-observed approval, denial, and failure frames are eligible. Its bounded local display and
click receipts never satisfy Core delivery, approval, or execution evidence. Organization-managed
A2/A4 routes remain separate channel-as-audience bindings. The server-side console-inbox A2
channel is a separate Core adapter that stores bounded alert records for the Console and drops message links, because the https-only link rule
belongs to external channels; see
Console inbox.
5. Channel Interfaces (contracts)
Section titled “5. Channel Interfaces (contracts)”-
A2/A4:
NotificationChannel.send(NotificationMessage) -> DeliveryReceipt.NotificationMessage.categoryis a semantic route key;trust_tiercarries A1-A4. -
A1:
HilChannel.send(HilApprovalRequest) -> HilApprovalReceiptandpoll(receipt) -> HilResponse. -
A3:
ConversationChannelAdapter.receive() -> InboundTurnandsend(OutboundResponse) -> ChannelDeliveryReceipt. -
Adapters MUST NOT authorize the decision themselves.
HilChannel.pollreturns the raw response; the core router hands it tofdai-api, which is the sole authority (identity re-verify, replay check, no-self-approval). -
Adapters MUST re-scan the message body for known secret patterns (same regex set used by the CI secret scanner) before dispatching, as a last-line defense.
-
Adapters MUST implement idempotent
send: a re-issued send with the samecorrelation_id + audit_id + categoryMUST NOT create a duplicate post. -
Slack A1 outbound is not an approval. The dedicated
HilChanneluses a fixed HTTPSchat.postMessageendpoint, a protected bot token, a configured channel id, and a shared HTTP client. It sends bounded Block Kit withfdai_hil_approveandfdai_hil_rejectbuttons whose only value is the parked approval id. The card includes an opaque dispatch id but no browser link, resource detail, action hash, role, or actor claim. A click starts a signed interaction, not a decision; the Operator receiver derives the actor from the verified Slack envelope and returns a server-origin Console handoff URL for fresh Entra reauthentication. The browser cannot supply approval authority. Acceptance requires Slackok=true, a bounded message timestamp, and the configured channel in the acknowledgement.pollalways returnsPENDING. The Core-owned request outbox persists an immutable dispatch identity and exact rendered-payload digest, including both button targets and the reminder dispatch id, with audit before HTTP. Atomic claims cover initial, grouped, and distinctly identified reminder posts. Only a matching provider acceptance receipt closes delivery; a lost acknowledgement or an expired in-flight claim is durablyunknownand held across restart without a blind repost. A queued, never-attempted request may be dispatched after restart only after rechecking the original pending park and expiry. No authoritative Slack readback or live authenticated Slack-to-Entra browser receipt exists under issue #943. Configuration or dispatch alone never raises approval or execution authority. -
A2/A4 capability-state, pre-render presentation, observation mode delivery, and provider-native rich card payloads are owned in detail by Multi-channel notification delivery § 8.
-
A1 decision return: The exact Teams and Slack transport, actor, context, workflow-routing, audit, and durable-delivery contract is owned by Operator approval callbacks.
5.1 Audience Derivation (channel-as-audience)
Section titled “5.1 Audience Derivation (channel-as-audience)”Recipient lists are not derived per-user by the router. Each channel is an audience, and membership is managed outside the control plane - typically by binding the channel to an Entra security group.
- Default (Option A): a Teams channel/DL is created as a group-connected team
backed by an
aw-*Entra security group. Membership syncs automatically from Entra (“Owner adds a person toaw-approversin the Portal” → they immediately see the next digest and every A1/A2/A3 post). This keeps administration in one surface (user-rbac-and-identity.md §4.2). - In-message
@mentionsare the declared design for calling out artifact-owners inside a channel post (e.g. the requester of an expiring exemption), derived from artifact metadata (requested_by, PR author, rule author) already carried in the audit stream - no Graph lookup at digest time - but the matrix loader does not yet enforce this mode. - Role-derived direct messaging is the declared design for break-glass usage summary only (small, time-critical audience where a channel post is not enough); every other A4 digest stays channel-only, and this mode too is not yet enforced by the matrix loader.
Allowed audience modes for a digest entry:
| Mode | Meaning | Where allowed |
|---|---|---|
channel: <id> | post to a channel/DL; membership managed via Entra group binding | A2, A3, A4 (default) |
mention-artifact-owner | additive: @mention the artifact’s owner inside a channel post | Declared design; not yet enforced by the matrix loader |
role-dm: <RoleName> | Graph-lookup members of aw-<role>, DM each | Declared design; not yet enforced by the matrix loader |
5.2 Proactive Stakeholder Briefing (A4 synthesis)
Section titled “5.2 Proactive Stakeholder Briefing (A4 synthesis)”An organization keeps a human who writes the periodic operations summary for
leadership - “here is what happened this window, what we did about it, and
where the risk is.” core/notifications/briefing.py
(StakeholderBriefingComposer) synthesizes that A4 digest deterministically
from aggregated operational counts (incident tallies by severity, action
outcomes split auto / human approval / rolled-back / observation mode-only, cost run-rate delta and
drivers, forward-looking forecast risks, guard-metric breaches) - not from
per-event noise.
- Fail-closed, no fabrication. The composer sources every figure from the
audit log / KPI telemetry the caller supplies and asserts nothing it was not
given. A window with no actionable activity renders an explicit “No
significant operational activity” headline and
has_significant_activity = False, so the caller suppresses the send rather than emailing leadership to say nothing happened. A sub-1% cost wobble with nothing else is treated as noise, not a briefing. - Guard breaches escalate. Guard-metric breaches (the one thing leadership
must never miss) are surfaced as explicit
escalationson the result so the caller can route them at a higher trust tier, in addition to appearing in the briefing body (goals-and-metrics.md). - Pure and delivery-agnostic. The composer holds no vendor knowledge and
never dispatches; it returns a
StakeholderBriefing(markdown body plus per-section payload) that the caller hands to the router in §6 for A4 delivery through the audience modes above. Same input, same briefing.
6. Routing Policy (config-driven)
Section titled “6. Routing Policy (config-driven)”Routing is declarative config, evaluated by the channel-router. Adding, replacing, or reordering channels is a config change, never a code change.
Config location: outbound routing lives in
config/notifications-matrix.yaml. Routing changes
receive governance review, with Owner-tier review for A1 routes. Only a change that can alter
A1 routing needs the trusted identity attestation; adding or changing an A2 to A4 route follows
normal review (Rule Governance).
The router refuses to deliver an A1 message through a non-A1 route and escalates it to the human approval
sink, so a new route can’t capture decision-bearing traffic. Conversation channel enablement
uses its separate environment/configuration contract.
matrix: version: 1 default_route: hil_approval routes: hil_approval: trust_tier: a1_hil_approval primary: teams-hil-prd fallback: [teams-hil-standby, slack-hil-prd] on_all_fail: hil_escalate operational_alert: trust_tier: a2_operational_alert delivery_mode: fanout channels: [teams-ops-prd, pagerduty-primary, email-oncall] on_all_fail: hil_escalate digest_shadow_accuracy_daily: trust_tier: a4_digest delivery_mode: fanout channels: [teams-hil-prd, email-governance] on_all_fail: hil_escalateRouter rules (MUST)
- Category ⊆ channel.categories - the router refuses to send a message to a channel whose declared categories do not include the message’s category. Startup config validation rejects a routing entry that pairs a channel with a disallowed category (deny-by-default; fail fast).
- Trust preservation on fallback - A1 primary → A1 fallback only. Downgrading to a lower trust level on fallback is a config-load error.
- Incident delivery readiness - A non-local control-plane runtime fails startup when the
operational_alertroute has no registered channel that supports its A2 trust tier. The explicit local Azure CLI profile may start without an external adapter, but it emits a structurednotification_route_unavailablelifecycle record atINFOand keeps fail-closed human approval escalation. - Incident severity escalation - A committed monotonic severity increase emits one A2
severity_changednotice. Its stable audit id deduplicates immediate delivery and startup replay. role-dmandmention-artifact-ownerare declared design, not yet enforced. The matrix loader (core/notifications/matrix.py) validates declared channels, trust tiers, andon_all_fail/default_routestructure today; arole-dmdeny-list limited tobreak_glass_usage_summaryand amention-artifact-ownermetadata-field check remain open follow-up work, not current config-load behavior.- Bounded retries - each adapter declares its own retry budget; router escalates to
the next channel or to
on_all_failon exhaustion. - Durable A1 decisions -
fdai-apirecords the normalized approver, decision, and receipt before event-bus publication. Publication uses a configured attempt count, per-attempt timeout, and bounded backoff. Success checkpointsdelivered; a resolved item leaves the pending queue. A retry by the same actor and decision resumes an undelivered receipt or returns the delivered receipt without republishing. A different actor or decision is a conflict. Startup and periodic recovery drain undelivered receipts without another human action. Delivery attempts persist and becomeabandonedat the configured ceiling; terminal delivery state never regresses.HilDecisionRecoveryConfigsets the recovery interval, per-attempt publish timeout, and delivery-attempt ceiling as constructor defaults; the recovery loop has no environment-variable override wiring in production composition today. - Rate policy stays deployment-owned - tenant-specific approver rate, quiet-hour, and fatigue limits belong in authenticated ingress and routing configuration. They never weaken registry idempotency, expiry, quorum, or the no-self-approval checks.
- Approval load plans never drop work - every A1 request is durably parked before a versioned
load policy (
config/approval-load.yaml, optionally selected byFDAI_APPROVAL_LOAD_POLICY) choosessend_now,deferred, orgroupeddelivery. Critical severities always bypass quiet hours and grouping. Non-critical requests may share one deterministic action and assignee group window or wait until quiet hours end, but every member remains independently visible and decidable in the approval queue. An assignee fatigue ceiling changes a group’s notification mode; it never blocks parking or implies approval. At the group-window or quiet-hour boundary, the runtime claims one stable group dispatch id and sends one anchor digest with the current pending member count. A deferred group and its members share that claim, so they cannot produce duplicate initial cards across replicas. Initial delivery and reminders use distinct metadata and audit kinds. - Reminders are bounded attempts - the policy declares deterministic offsets before the original approval expiry. A runtime worker claims each reminder id once across replicas and attempts the existing A1 channel. Success or failure is audited; neither extends TTL, removes the pending item, or retries beyond the declared reminder count. Policy simulation must prove that critical requests are never deferred or grouped and that no input request disappears.
- TTL fail-closed without expiry spam - the runtime worker atomically changes an expired
pending park to terminal
timeoutand appends one audit entry across replicas. Late callbacks return the existing timeout and never execute. The load controller does not emit one A2 message per expired item; a notification layer can aggregate the audit signals into a bounded A2/A4 summary (security-and-identity.md).
7. Channel-Specific Notes
Section titled “7. Channel-Specific Notes”| Channel | Notes |
|---|---|
| Teams | Adaptive Cards for A1; keep the OAuth scope set minimal (ChannelMessage.Send.Group + bot signaling). SSO + OBO already covered in user-rbac-and-identity.md §10.4. Configure FDAI_TEAMS_APPROVAL_TEAM_ID, FDAI_TEAMS_APPROVAL_CHANNEL_ID, the HTTPS FDAI_TEAMS_APPROVAL_ACTIVITY_URL, and the dedicated FDAI_TEAMS_BOT_MI_CLIENT_ID for the group-connected team backed by an aw-* Entra security group. Core sends through this Bot identity, never the executor identity, and places the resulting teams:<team-id>:<channel-id> audience on the card; Operator validates the same value. An outbound-only notification webhook, including a Teams Workflows trigger, is not an A1 transport because it cannot deliver an Action.Execute callback or return an authenticated approver; binding parsing refuses an a1_hil_approval claim on any notification binding. The receiver additionally needs FDAI_TEAMS_APPLICATION_ID, FDAI_TEAMS_TENANT_ID, FDAI_TEAMS_ALLOWED_SERVICE_URLS_JSON, FDAI_TEAMS_JWKS_URL, FDAI_TEAMS_PRINCIPAL_MAP_JSON, the shared callback secret, and a configured decision requiring human approval topic plus durable outbox; Teams A1 stays closed until all inputs are present. |
| Slack | Block Kit for A2/A3 and bounded A1 buttons; chat:write requires FDAI_SLACK_APPROVAL_API_URL (the exact https://slack.com/api/chat.postMessage URL), FDAI_SLACK_APPROVAL_CHANNEL_ID, and protected FDAI_SLACK_APPROVAL_BOT_TOKEN. Slack-only outbound configuration needs no Teams Bot identity. Posting and clicking do not approve. The signed interaction receiver requires a configured workspace, userId-to-OID map, and a single approved Console origin. The browser decision requires a fresh signed Entra API-token auth_time. Core supervises the Slack A1 request outbox separately from report/review outboxes, so a delivery retry cannot become an approval. Missing bindings hold the decision; deployed click-to-decision evidence remains open. |
Send-only through Azure Communication Services Email. Never include an approval link; digest and alert only. The adapter sends plainText for every message and adds bounded HTML when notice_kind=opened. That incident template uses only incident id, state, severity, opened time, aggregate member count, assignment state, audit_id, and an HTTPS Console link. It never renders correlation keys, resource payload, actor identity, or the free-form reason. Terraform provisions an Azure-managed sender domain and a dedicated notification managed identity scoped to the Communication Services resource. FDAI_CONSOLE_BASE_URL supplies the Console origin; when it is absent or the resulting link is not absolute HTTPS, the renderer omits the CTA. The adapter requests a short-lived https://communication.azure.com/.default token, waits for the provider operation to reach Succeeded, and records the provider message id. Settings > Integrations fetches the same renderer through an authenticated GET using synthetic placeholders only. Recommended recipient: an Entra dynamic distribution group mirroring aw-approvers / aw-owners. | |
| Generic webhook | HMAC-SHA256 signature, monotonic timestamp, single-use nonce. Receiver failures never block; core retries per adapter policy and moves on. |
| PagerDuty / Opsgenie | Deduplication key = the observability correlation id so a burst collapses. Runbook URL is required in every alert. |
| SMS | Payload restricted to <severity> <audit_id> <short-url-to-runbook>. No secrets, no customer names, no free-form text. Break-glass reachability primarily. |
| Direct Line-compatible custom channel | Send-only A2/A4 activities for a deployment-owned relay. Rehearsal is the default mode, relay acceptance never counts as delivery, and approvals, commands, and replies are unavailable. Upstream validates the contract with a loopback example CLI, not an external Direct Line environment. See Multi-channel notification delivery § 9. |
7.1 Teams readiness is reported per category
Section titled “7.1 Teams readiness is reported per category”Teams carries four different contracts with different owners. The A1 contract has separate send and callback owners, so Settings > Integrations reports five independent rows instead of one aggregate “ChatOps” status. Readiness means configuration completeness observed in one runtime; it is never a provider health claim.
| Row | Owning runtime | Ready when |
|---|---|---|
teams-a1-approval-send | Core control plane | The Bot activity endpoint plus the group-connected team and channel are all present |
teams-a1-approval-callback | Operator Service | The Bot application, tenant, JWKS, allowed service URLs, principal map, callback secret, decision requiring human approval topic, and durable outbox are all present and structurally valid |
teams-a2-operational-alert | Core control plane | A Teams Workflows binding declaring a2_operational_alert is activated and its endpoint is not the seeded placeholder |
teams-a4-digest | Core control plane | The same, for a4_digest |
teams-a3-conversation | Operator Service | The conversation channel edge enables Teams and its identity inputs are present |
Partial configuration is incomplete, not degraded: a half-configured approval path leaves
approvals unavailable rather than silently downgraded. Each row also carries its owning runtime and
an observed flag, because a projection produced by one process cannot see another process’s
inputs. An unobserved row reports “not observed here” rather than claiming “not configured”, so an
operator repairs the runtime that actually owns the prerequisite.
8. Fallback and Kill-Switch Interaction
Section titled “8. Fallback and Kill-Switch Interaction”- The global kill-switch halts every A1 dispatch immediately and re-queues open A1 requests; kill-switch state itself is announced via A2 on every operational channel.
- If all A2 channels are down, adapter health telemetry still lands in observability and appears in the console; the kill-switch remains operable through its dedicated break-glass path (security-and-identity.md).
- Adapter unhealth is itself an A2 signal - a Teams outage that stops A1 delivery pages the operational lane through the fallback channel.
9. Fork vs Upstream Split
Section titled “9. Fork vs Upstream Split”| Item | Upstream (this repo) | Fork |
|---|---|---|
| Three provider contracts plus their message/receipt types | ✓ | - |
| Teams adapters | A1 and A2/A4 implemented; A3 planned | tenant / group-connected team binding |
| Slack A1 callback and outbound | locally implemented, including signed browser handoff and durable button delivery; default-disabled without complete bindings | workspace credentials, one approved Console origin, API-token auth_time, and userId↔OID mapping (required) |
| ACS Email adapter | ✓ (A2/A4, managed identity, final-status polling) | recipient binding + enablement |
| Webhook / PagerDuty / SMS adapters | ✓ (concrete delivery adapters) | credentials + enablement |
| Direct Line-compatible custom-channel adapter | ✓ (A2/A4, rehearsal default, acceptance-only receipt, loopback example CLI) | production relay, conversation, secret, and governed activation |
| Routing-config schema + startup validation | ✓ | deployment-specific bindings/overlays |
human approval escalation sink (on_all_fail fail-safe queue) | ✓ (StateStoreHilEscalationSink - StateStore-backed, tenant-agnostic) | own queue backend (optional) |
| Seven default digests + audience derivation rules | ✓ | cron timezone, channel ids, on/off per digest |
| Secret-scan regex set (reused by adapters) | ✓ | extend patterns if needed |
| Slack userId ↔ Entra OID mapping interface | ✓ | mapping data (mandatory for P1 A1) |
| Digest content templates | ✓ (generic) | branding / localization |
10. Open Decisions
Section titled “10. Open Decisions”- Adapter-health alert thresholds and dedupe windows.
- Incoming webhook scope - only authenticated
TypedWebhookMappingcan publish an allowlisted event/agent target; payloads cannot select commands, targets, or sessions. - The
mention-artifact-ownerbehavior when the artifact owner is a guest user (mention still resolves in Teams, but should the digest suppress or route differently to reduce information leakage?). -
kpi_and_cost_monthlyGitHub-Issue archive: destination repo/path (defaults to the catalog-as-code repo,docs/kpi-archive/).
11. Localization (L2)
Section titled “11. Localization (L2)”Notifications are an L2 product surface (see language.instructions.md): the source strings are English, and a channel MAY render them in another locale.
- How it renders (Option C).
corenever bakes a final localized string. EveryNotificationMessagecarries atemplate_keyplus typedparams; the router renderstitle/body_markdownfrom the catalog (services/core-control-plane/src/fdai/core/notifications/messages.{en,ko}.json) in the destination channel’s locale, just beforesend. Adapters are untouched - they still consumetitle/body_markdown. - Only labels localize. The L0 values (decision word, rule ids, resource type, mode) are substituted verbatim in every language, so the machine-readable data is identical. The audit entry always uses the English message, so replay and correlation stay language-neutral.
- Mandatory English fallback. A missing locale key or field renders the English source; a missing English key renders the key itself (never a blank).
- Locale is a channel property. Notifications fan out, so the locale is set
per channel in
config/notifications-matrix.yamlundermatrix.channels(<channel-id>: { locale: ko }), not per operator. A channel without an entry renders in English.
Related docs
Section titled “Related docs”| To learn about | Read |
|---|---|
| Delivery status and remaining work | Implementation ledger |