Skip to content

ACR image retention — keep newest 3, prune older on deploy/upgrade

Motivation

Every control-plane deploy pushes a fresh image tag per sidecar repository (elb-api, elb-frontend, elb-terminal). Both the in-app self-upgrade (Settings → Update) and scripts/dev/quick-deploy.sh therefore accumulate one manifest per run in the platform Azure Container Registry (ACR) forever, growing registry storage unbounded. There was no automatic cleanup — old, unreferenced images piled up.

User-facing change

Old images are now pruned automatically, keeping only the newest 3 manifests per control-plane repository (configurable). The prune runs at the two moments the user named:

  • In-app upgrade — after an upgrade reaches succeeded, the reconciler runs a best-effort retention sweep. The now-running images and the rollback target are protected so a subsequent rollback is never starved of its snapshot. Tunable via UPGRADE_ACR_KEEP_IMAGES (default 3).
  • quick-deploy.sh — after a successful build, the script deletes older manifests for the repositories it just deployed, while the ACR firewall is still open (steady-state ACR is publicNetworkAccess: Disabled, so the prune runs between acr_ensure_build_access and acr_restore_build_access). Tunable via ELB_ACR_KEEP_IMAGES (default 3); skip with --no-prune or ELB_SKIP_ACR_PRUNE=1. Skipped automatically on --no-build (no fresh image was pushed that run). The current template and every active revision are inspected before deletion; their image digests remain protected even when build-only runs have moved them outside the newest-N window.

Both paths are best-effort: a missing AcrDelete/Contributor permission, a transient registry error, or a repository with ≤ keep manifests is a no-op and never fails the deploy/upgrade. The newest keep manifests (by last-update time) are never deleted. Quick deploy also fails closed and skips the sweep when live references cannot be read or resolved, so retention cannot delete an image needed to start a revision. See the live-manifest protection follow-up.

API / IaC diff summary

  • New service module api/services/upgrade/acr_retention.py — prune_repository, prune_control_plane_images, keep_count. Data-plane prune via azure-containerregistry (list_manifest_properties + delete_manifest); reuses acr_inventory for client construction + ref parsing.
  • api/tasks/upgrade/reconciler.py — new _prune_acr_after_success(row) helper called on both the single-mode and blue/green succeeded transitions. Fully best-effort; never raises.
  • scripts/dev/quick-deploy.sh — new acr_prune_repo_keep_recent / acr_prune_targets helpers, --no-prune flag, ELB_ACR_KEEP_IMAGES / ELB_SKIP_ACR_PRUNE env knobs; invoked before the final "Done" in the all and single-sidecar deploy paths.
  • No Bicep / RBAC change. The shared user-assigned MI already holds Contributor on the registry (infra/modules/acr.bicep). Caveat: the data-plane delete_manifest content-delete action maps to the AcrDelete role; if a deployment's MI lacks it the in-app prune logs forbidden (MI needs AcrDelete) per repo and the upgrade still succeeds (the registry simply keeps accumulating until AcrDelete is granted as an additive phase-1 role assignment per charter §12a Rule 1).

Validation evidence

  • uv run pytest -q api/tests/test_upgrade_acr_retention.py → 13 passed (keep-newest-N, protected tag/digest outside window, env override, forbidden delete best-effort, list-not-found no-op, control-plane orchestration).
  • uv run pytest -q api/tests/test_upgrade_task.py api/tests/test_upgrade_chaos.py → 62 passed (success hook does not break the state machine).
  • uv run pytest -q api/tests/test_upgrade_routes.py api/tests/test_upgrade_bluegreen.py api/tests/test_upgrade_acr_inventory.py → 73 passed (blue/green confirming → succeeded prune hook safe).
  • uv run ruff check clean on the changed Python files.
  • bash -n scripts/dev/quick-deploy.sh → syntax OK.