Skip to content

Fix prepare-db verify false-failures under azcopy 10.32

Motivation

The nt prepare-db AKS Job (prepare-db-nt-260602010502) reported Failed 0/10 (BackoffLimitExceeded, 12 pods in Error) while it looked, from the dashboard, like it was "still downloading". Each shard pod copied files 1-9 cleanly, then on the first integrity-sampled file (every VERIFY_EVERY_N-th, i.e. file 10 — a ~3 GB nt_euk.021.nsq volume) logged:

ERROR size mismatch .../nt_euk.021.nsq exp=2999987448 got=2.79 GiB
DONE shard=04 ok=9 fail=1 skip=0

Two distinct bugs combined:

  1. Human-readable size compare. azcopy auto-upgraded to 10.32.4, whose azcopy list --output-type=json reports ContentLength as a human string ("2.79 GiB") instead of a raw integer. The verify step compared that string byte-for-byte against the raw NCBI Content-Length (2999987448), so every multi-GB sampled file false-failed — even though 2999987448 bytes == 2.79 GiB and the upload was correct. On a false mismatch the script ran azcopy remove, deleting the healthy blob.

  2. Loop input drained by azcopy remove. The shard loop read the file list on stdin (done < "$FILE_LIST"). azcopy remove was invoked without an stdin redirect and azcopy drains fd 0, so the first remove swallowed the remaining file-list lines and the loop ended after one file — hence ok=9 fail=1 and an immediate DONE instead of continuing through the shard.

User-facing change

  • prepare-db downloads no longer false-fail (and no longer delete correct blobs) on integrity-sampled multi-GB files under azcopy >= 10.32.
  • A shard now processes its entire file list even when a genuine mismatch triggers an azcopy remove.
  • Genuine truncations / HTML error bodies are still caught (they differ from the expected size by far more than the tolerance).

Code change summary

api/services/k8s/prepare_db_jobs.py (PREPARE_DB_AKS_SCRIPT):

  • blob_content_length() parser now normalizes ContentLength to an integer byte count whether azcopy emits a raw integer (<= 10.31) or a human string ("2.79 GiB", "512 B", …, binary and decimal units), returning PARSE_FAIL on shapes it cannot interpret.
  • The post-upload verify replaces exact string equality with a 1% / 1 KiB tolerance byte compare (abs(up - exp) <= max(1024, exp // 100)) to absorb the ~3-significant-figure precision loss of the human format.
  • The shard loop reads the file list on fd 3 (read -r KEY <&3 / done 3< "$FILE_LIST") and azcopy remove gets `