Skip to content

Fix the Service Bus bridge claim/release etag wedge

Motivation

A producer reported that requests sent to the Service Bus request queue sometimes never received a completion event. A production audit of ca-elb-dashboard (Log Analytics workspace 648cd0d4-…) confirmed a real, narrow defect.

The completion path itself was healthy — terminal events were being published (241 / 84 / 109 / 26 on 08-02 → 08-05), the durable response outbox had zero pending rows, and the elastic-blast-completions topic's single default subscription (TrueRuleFilter) held zero active and zero dead-lettered messages. The failure was one step earlier, in the drain's single-writer reservation.

_claim_table and _release_table read the optimistic-concurrency etag as dict(entity).get("odata.etag"). azure-data-tables moves every odata.* field of the wire payload into TableEntity.metadata during deserialization, so that expression is always None, and the SDK then raises ValueError: IfNotModified must be specified with etag.

The resulting wedge, reproduced exactly in the logs for wf3:718:exclusive:E3L:72551461:

  1. 06:15:31Z — the drain wins the claim, then defers the submit because the sibling OpenAPI plane is not ready (openapi_not_ready). The rollback release_bridge raises, is swallowed at DEBUG, and the placeholder row survives.
  2. 06:16:52Z – 06:18:07Z — the drain's own retry-… redeliveries of the same request see the leftover reservation and log claim_contended (message ids confirm a single producer send, delivery_count=0; these are not producer duplicates).
  3. 06:19:45Z – 06:20:00Z — past the 180 s stale threshold the steal path raises ValueError out of claim_bridge, so _safe_drain_handler abandons the message.
  4. The request is never submitted: no job, no queued ack, no completion event. The unconfirmed bridge row lingers as an active bridge (visible as the permanent publish_transitions floor of scanned: 14 / published: 0) until the 7-day bridge_unconfirmed_timeout finally emits a terminal failure.

Measured blast radius over 7 days: 126 ValueError occurrences and 13 correlation ids that hit claim contended and never reached stage=accepted (9 real WF3 requests, 4 load-test messages). On 2026-08-05 (KST) 52 of 54 requests completed normally with completion_published; the only 2 failures were this wedge.

User-facing change

A request whose submit is deferred (most commonly while AKS is stopped or warming up) is now correctly rolled back and re-submitted on the next redelivery, instead of being wedged permanently. From the producer's point of view, the "request accepted but no completion event, ever" case is gone.

Change summary

  • api/services/service_bus_tracking.py
  • New _entity_etag(entity) helper: reads metadata["etag"] first and keeps the mapping key only as a fallback for plain-dict callers/fixtures. Returns "" when no etag is available rather than handing None to the SDK.
  • _claim_table keeps the TableEntity (no longer dict()s it before extracting the etag) and passes the real etag to the conditional steal. When no etag is available it now refuses the steal (returns False, logged at WARNING) — deferring a delivery is safe, an unguarded steal would risk a duplicate BLAST submit.
  • _release_table likewise uses the real etag for the conditional delete, and skips (with a WARNING) when none is available.
  • _release_table's catch-all moved from DEBUG to WARNING. The DEBUG level is what hid this failure in production; the release stays best-effort by contract but is no longer silent.
  • Module context header gained the etag contract under Risky contracts.
  • No IaC, route, schema, or frontend change. No new dependency.

Audited every other MatchConditions.IfNotModified call site — auto_stop.py, auto_warmup.py, upgrade/state.py, storage/prepare_db_metadata.py, and scripts/dev/openapi-overlays/eta.py all already read the etag correctly. service_bus_tracking.py was the only affected module.

Hardening round (design critique follow-ups)

Restoring the steal path also made three latent defects reachable for the first time, so they are fixed in the same change:

  • A claim-store outage no longer burns the broker delivery budget. api/tasks/servicebus/tasks.py now wraps claim_bridge: any unexpected exception defers with the new claim_unavailable error code instead of escaping to _safe_drain_handler, which converts it to ABANDON. An ABANDON per tick during a Table outage consumes the queue's ~10 deliveries and dead-letters a perfectly valid request; the expiry-preserving RETRY keeps the retry budget intact and still never submits, so the single-writer guarantee holds. claim_unavailable is a new error_code — it is deliberately distinct from claim_contended so an outage is not misread as normal contention, and no existing consumer switches on either value (both are log/telemetry-only dimensions).
  • Celery soft time limits are never swallowed. Both the new claim guard and _release_table's catch-all re-raise SoftTimeLimitExceeded, matching the repo-wide contract already used by service_bus_health.py, aks/execution_admission.py, feature_events.py, and k8s/warmup_status.py.
  • A benign delete race stays quiet. _release_table's conditional delete now swallows ResourceNotFoundError alongside ResourceModifiedError. Without this, the freshly-raised WARNING catch-all would print a traceback every time a concurrent release removed the row first — training operators to ignore the very line that makes this class of bug findable.
  • The stale-claim threshold has a real safety floor. _CLAIM_STALE_FLOOR_SECONDS is 120 s, above external_blast._DEFAULT_TIMEOUT_SECONDS (90 s), replacing the previous 30 s floor. Stealing a reservation whose submit is still in flight lets two workers submit the same correlation id — a duplicate BLAST run on the cluster. That was unreachable while the steal always raised; it is reachable now. The default stays 180 s and SERVICEBUS_CLAIM_STALE_SECONDS is not set on any deployment, so no environment changes behaviour.

Residual, accepted: a create whose server-side write succeeds but whose response fails leaves a phantom reservation. It is bounded — the next redelivery contends, and once the row passes the stale threshold the (now working) steal path takes it and submits. That bounded self-heal is exactly the property the etag fix restored.

Validation

  • uv run pytest -q api/tests/test_service_bus_tracking.py — 22 passed. Table-backend tests drive _claim_table / _release_table through a fake TableClient that reproduces the SDK's own precondition (raise ValueError("IfNotModified must be specified with etag.") on a falsy etag) against a real TableEntity whose etag lives in metadata, so the pre-fix code fails them:
  • test_claim_table_steals_stale_reservation_using_metadata_etag
  • test_claim_table_refuses_steal_when_etag_is_unavailable
  • test_claim_table_does_not_steal_a_confirmed_row
  • test_release_table_deletes_unconfirmed_row_using_metadata_etag
  • test_release_table_never_deletes_a_confirmed_row
  • test_entity_etag_prefers_metadata_over_mapping_key
  • test_release_table_is_quiet_when_the_row_vanishes_mid_delete
  • test_release_table_logs_unexpected_failures_at_warning
  • test_release_table_propagates_celery_soft_time_limit
  • test_stale_claim_threshold_outlives_a_slow_sibling_submit
  • uv run pytest -q api/tests/test_servicebus_tasks.py — includes test_claim_store_outage_defers_instead_of_burning_delivery_count and test_claim_soft_time_limit_is_never_converted_into_a_defer.
  • uv run pytest -q api/tests — 4959 passed, 3 skipped.
  • uv run ruff check api — clean. mypy on the changed modules reports no new findings (only pre-existing ones on untouched lines).

Live end-to-end run on the deployed revision

Deployed as elb-api:20260805121915 (sha256:eccadeb…) onto api / worker / beat, revision ca-elb-dashboard--0000265 (RunningAtMaxScale, Healthy). One real request was sent from the Service Bus Playground with the AKS cluster stopped — the condition the wedge needed:

Time (UTC) Event
12:33:49 enqueued — corr=42b3016d75a740d99f67cc9b501854ec, request_id=verify-etagfix-20260805, db=core_nt (sharded), blastn, multi-token outfmt 7, resource_profile=core_nt_safe
12:33:5x queue-arrival auto-start fired; elb-cluster-01 → Starting
12:41:32 received (first delivery, delivery_count=0)
12:41:44 row_created → routed → submitted, job_id=68b0010d2e26
12:41:58 running transition published (publish_transitions published=2, errors=0)
12:56:19 succeeded — terminal completion event published (published=1 finished=1 errors=0)

End state: request queue active=0 / dlq=0, completion subscription default active=0 / dlq=0, job completed.

Error signals across the whole window: IfNotModified 0, claim contended 0, claim_unavailable 0, release_bridge (table) failed 0 (previously ~20 IfNotModified per day).

Coverage caveat, stated plainly: because queue-arrival auto-start defers the whole drain tick until the sibling plane is ready, this run never took a post-claim deferral, so the repaired _release_table / steal branches were not themselves executed live. They are covered by the unit tests above, and the deployed image digest matches the tested code. The standing production signal to watch is that IfNotModified occurrences stay at zero.

Operational notes

  • The 9 already-lost requests do not self-heal: their messages left the queue before the fix. The producer must resend those correlation ids.
  • The pre-existing unconfirmed bridge rows stay active until their 7-day bridge_unconfirmed_timeout, which emits a terminal failure event — late, but not silent.
  • This module runs in the worker sidecar, so the fix reaches production with the next image rollout; there is no template or sidecar-layout change.