AKS idle auto-stop no longer stops a freshly started cluster¶
Motivation¶
A user started an AKS cluster from the dashboard and it stopped again within a
few minutes. Production logs for elb-cluster-02 (idle window 15 min) showed the
idle auto-stop loop repeatedly stopping the cluster right after each start
(stops at 06:48, 08:20, 13:21) and at 13:21:22 a Celery task ERROR:
followed by a manual restart at 13:27. Two distinct defects were confirmed:
- A cluster start did not reset the idle clock. The evaluator anchors the
idle deadline on the most recent BLAST job timestamp (or
pref.created_at). When the last job was older thanidle_minutes, the very next beat tick (≤ 5 min) computed a deadline already in the past and decidedstop— so the cluster was torn down moments after the user started it. - Stopping mid-start crashed the task. AKS
power_state.codereports"Running"the instant a start LRO begins, whileprovisioning_stateis still"Starting". The evaluator's only power gate saw"Running", issued a stop, and ARM rejected it withOperationNotAllowed— surfacing as a Celery task ERROR.
User-facing change¶
- Starting a cluster (via
start_aks) now grants it a fullidle_minutesgrace window before the idle auto-stop loop may stop it again. No more "started, then immediately stopped" surprises. - The auto-stop act task no longer attempts to stop a cluster whose
provisioning_stateis anything other thanSucceeded(e.g.Starting,Updating), removing theOperationNotAllowederror class from the worker.
No SPA / API response shape changed — the new preference field is internal and is
not projected by the public _PUBLIC_PREF_FIELDS allowlist on
/api/aks/*/autostop.
API / IaC diff summary¶
api/services/auto_stop.pyAutoStopPreferencegainslast_started_at: str = ""(round-trips throughto_dict/from_dict).- New CAS-safe helper
mark_auto_stop_started(subscription_id, resource_group, cluster_name) -> AutoStopPreference | None— stampslast_started_at/updated_at; no-op (returnsNone) when no preference row exists. api/services/auto_stop_evaluator.pyevaluate_cluster(...)gains an optionalprovisioning_state: str = ""keyword and keeps the cluster (reason="provisioning:<state>") when it is set and notsucceeded.- The idle anchor now also considers
pref.last_started_at, so a recent start extends the deadline even when no jobs are observed. api/services/cluster_health.pyClusterHealthgainsprovisioning_state: str | None, populated from the cluster meta snapshot on every return path.api/tasks/azure/idle_autostop.py- New
_provisioning_state(pref)helper (shares the 90 s cluster-health cache, non-fatal). The act task passes it intoevaluate_cluster. The beat decide path is intentionally unchanged. api/tasks/azure/lifecycle.pystart_akscallsmark_auto_stop_started(...)before issuing the start LRO, stamping the anchor as early as possible to close the decide-vs-act race.
No Bicep / IaC change.
Validation evidence¶
uv run pytest -q api/tests→ 2439 passed, 3 skipped (one pre-existing flakytest_terminal_exec.py::test_run_truncates_stdout_above_capunder parallelism; passes in isolation, unrelated to this change).uv run ruff check api→ All checks passed.- New / updated tests:
api/tests/test_auto_stop.py—mark_auto_stop_startedstamps + persists, no-op without a row,last_started_atdict round-trip.api/tests/test_auto_stop_evaluator.py— recent start resets the idle clock, stale start does not block stop, recent start anchors when no jobs observed, transitionalprovisioning_statekeeps the cluster,Succeededallows stop.api/tests/test_azure_tasks.py—start_aksstampslast_started_at.api/tests/test_cluster_health.py,api/tests/test_prepare_db_aks_route.py,api/tests/test_auto_stop_task.py— updated for the newprovisioning_statefield / parameter.
Per charter §13 this fix is validated by tests + reasoning and is not redeployed here; it lands on the next normal deploy.