Skip to content

SSE ticket failures return 204 (not 401) to break browser auto-retry storm

Motivation

App Insights Failures showed GET /api/monitor/logs/{container}/events returning 401 ×46 in 2 hours (≈96% of all failures), plus a smaller trail on /api/monitor/sidecars/events and the same pattern on /api/blast/logs/{job_id}/events. The endpoints themselves are healthy — the 401s are phantom failures generated by browsers' native EventSource auto-retry behavior after a stream drop:

  1. Frontend POSTs /{...}/ticket (MSAL bearer) → server stores a single-use ticket with a 30 s TTL.
  2. Frontend opens EventSource(/{...}/events?ticket=X) → server calls _consume_*_ticket() which pops the ticket immediately and opens the stream.
  3. Stream drops (ACA ingress idle ≈240 s, sidecar restart, network blip) → browser's native EventSource auto-retries the same URL → ticket was already popped → previous code returned 401 → the frontend's own onerror handler then closes the connection and reconnects with a fresh ticket.

Net effect: every drop emitted at least one phantom 401 that App Insights counted as a real Dependency failure. Six Live Wall tiles ×30 min idle pattern accumulated ~30+ noise events without any actual problem.

Per the HTML spec for Server-Sent Events:

Clients will reconnect if the connection is closed; a client can be told to stop reconnecting using the HTTP 204 No Content response code.

204 is the documented "stop retrying" signal. App Insights also does not count 2xx as failures.

User-facing change

  • App Insights Failures count for the three SSE endpoints drops from ~96% of all failures to near-zero. The dashboard's Failures blade now only shows real issues (transient /blast/databases 401 during MSAL refresh, real upstream /aks/openapi/proxy 404, etc.).
  • No SPA behavior change. The frontend's bounded retry path (scheduleReconnect in useSidecarLogs, useSidecarMetrics, useBlastJobLogStream) still fires on the EventSource error event and re-issues a fresh ticket. The native EventSource just stops hammering the consumed URL.

API / IaC diff summary

  • api/routes/monitor/logs.py
  • _consume_log_ticket(...) now returns _LogTicket | None (None for missing / invalid / expired) instead of raising 401.
  • logs_events(...) returns Response(status_code=204) when the ticket is None; otherwise opens the SSE stream as before.
  • Return annotation widened from StreamingResponse to Response.
  • api/routes/monitor/sidecars.py
  • Same conversion for _consume_sidecar_ticket(...) / sidecars_events(...). Removed now-unused HTTPException import.
  • api/routes/blast/logs.py
  • Same conversion for _consume_log_ticket(job_id, ...) / blast_job_logs_events(...). The previous 403 "ticket job mismatch" case is now also 204 — same auto-retry storm reasoning applies, and the mismatched ticket is still consumed (popped) so it cannot be replayed against another job.
  • No IaC change. No new dependency. No frontend change.

Validation evidence

  • uv run pytest -q api/tests/test_sidecar_logs.py api/tests/test_sidecars_events_route.py api/tests/test_blast_log_routes.py — 20 passed (12 new regression tests asserting 204 for missing / invalid / reused / expired / cross-job-mismatch tickets, plus the existing 8 tests still green).
  • uv run pytest -q api/tests — 1531 passed (1509 → 1531, +22 from this change combined with the earlier auto-warmup JobState seed change).
  • uv run ruff check on all touched files — clean.
  • cd web && npm run build — built clean.
  • cd web && npm test -- --run — 51 files / 383 tests passed.
  • Frontend consumers verified manually — every useSidecar* / useBlastJobLogStream hook's error event handler already closes the EventSource and runs its own bounded scheduleReconnect with a fresh ticket, so the 204 transition is transparent.

Why the other 1-off failures stay as-is

Investigated via direct App Insights query (requests | where success == false):

  • GET /api/blast/databases 401 — single hit in 2h with empty subscription_id/storage_account/resource_group. Duration 4ms = failed at the require_caller auth dep. This is the SPA racing MSAL token refresh on initial mount; converting this to 204 would mask real auth issues. Left as-is.
  • GET /api/aks/openapi/proxy?path=/v1/databases/core_nt 404 — single hit, upstream OpenAPI sidecar honestly returned 404 for a path that doesn't exist. Converting this to 204 would mask real API contract issues. Left as-is.