Serialize CompleteMultipartUpload across HA pods (#148)

Prod incident 2026-07-31: Scylla Manager backup on web-production-data/scylladb failed 6.090 GiB on a single node (10.43.221.216 / scylladb-eu-north-1-searchengine-rack-1) when rclone finalized one 6.1 GB main.companies SSTable. The scylla-manager-agent logged NoSuchUpload for upload id 2~TI-z2R_Pa-ncj6k5KVM7MzKwnnIiaUo.

HAProxy routed concurrent CompleteMultipartUpload requests for the same upload_id to different s3proxy pods. Each pod deleted/recovered upload state independently and both called upstream CompleteMultipartUpload, yielding This multipart completion is already in progress, premature UPLOAD_STATE_DELETED, and eventually NoSuchUpload. A s3proxy rollout (20→30 replicas) during the backup increased cross-pod state misses.

Changes

• CompleteUploadLock — Redis-backed (or in-memory fallback) per-upload lock so only one pod can finalize a multipart upload at a time.
• Idempotent complete — if metadata and assembled object already exist (peer pod finished first), return success without calling upstream again.
• handle_complete_multipart_upload runs entirely inside the lock.
• Unit tests for concurrent complete serialization, peer-finished idempotency, and lock timeout (SlowDown).

Deploy

Images are published automatically by CI on this tag:

• ghcr.io/serversidehannes/s3proxy-python:2026.8.1
• ghcr.io/serversidehannes/s3proxy-dashboard:2026.8.1
• Helm chart: oci://ghcr.io/serversidehannes/charts/s3proxy-python:2026.8.1

Full changelog: 2026.7.31...2026.8.1

Retry message-less 400 on UploadPart (#147)

Hetzner sheds load under concurrency by returning a bare 400 with an empty body, which botocore surfaces as InvalidArgument with Message=None. #144 made GatewayTimeout retryable but not this variant, so it raised on first sight and failed the whole multi-GB part — and through rclone's job error, the entire Scylla backup run.

All 4 occurrences captured on main.companies had Message=None, on legal 50 MiB non-final parts (parts 2–16, never part 1). A real InvalidArgument always names the offending argument, so the message-less form is congestion, not a malformed request.

Reproduced: 24 × 50 MiB parts at 16-way concurrency yielded 1 failure, surfaced as GatewayTimeout("The server did not respond in time") — the same congestion event with a body Hetzner managed to write. Latency degrades from p50 1.79s unloaded to p50 13.09s / max 52.32s under concurrency.

Changes

• is_retryable_source_error now treats InvalidArgument/BadRequest/400 with no message as retryable. Deliberately narrow — a genuine InvalidArgument with a reason still fails immediately, so a malformed request cannot retry forever.
• New UPLOAD_PART_RECOVERED — a part that only landed after N attempts, previously a silent success that hid how close a run ran to the edge.
• New UPLOAD_PART_GAVE_UP — carries attempts, max_attempts, classified_retryable, exhausted, part_size, elapsed_sec, AWS code/message. "Gave up after N transient failures" is now distinguishable from "rejected outright"; they were identical in UPLOAD_PART_CLIENT_ERROR.
• UPLOAD_PART_RETRY gains part_size, aws_error_code, max_attempts.

Still open

• ContentLengthError truncated reads continue at ~5–30/min (resume from #144 handles them, but the underlying congestion remains)
• Scylla Manager has no retry above the job level (worker_upload.go:213 retries only errJobNotFound), so any error escaping the proxy still costs a whole host's upload
• aws_error_message: null on a legal 50 MiB part is arguably a Hetzner-side bug worth reporting upstream

Full changelog: 2026.7.30...2026.7.31

fix: retry 504 GatewayTimeout and resume truncated source reads (#144)

Two independent Hetzner failure modes that took down the Scylla main.companies backup on 10 of 13 racks, plus a test-fidelity fix that let one of them hide.

1. Unretried 504 GatewayTimeout — 95% of failures

Census of the 288 fatal copy failures captured live during a failing run:

273 x  504 GatewayTimeout  on UploadPartCopy
 17 x  504 GatewayTimeout  on PutObject
  7 x  504 GatewayTimeout  on GetObject
  8 x  ContentLengthError (truncation)

copy.py already retries via is_retryable_source_error, but _RETRYABLE_S3_ERROR_CODES held 500, 502, 503 — and not 504. The smoking gun:

297 x  504 GatewayTimeout
  4 x  UPLOAD_PART_COPY_SEGMENT_RETRY

Reproduced: 572 UploadPartCopy calls at 8-way concurrency → 1 x 504 (0.17%), latency p50 1.14s / p90 3.31s / max 61.86s. Heavy-tailed latency means a 504 is transient congestion when Hetzner's own server-side copy exceeds an internal deadline.

At 0.17% per copy, a 4.7GB file (572 copies) has a ~63% chance of hitting one; 10 nodes concurrently approaches certainty.

botocore does retry 504, but all its attempts fire inside the same congestion window and exhaust — "reached max retries: 3" x297. Retrying with exponential backoff gives the backend time to clear. UploadPartCopy is idempotent for a given PartNumber+range.

2. Truncated source reads

Hetzner ends a response mid-body with a clean TCP FIN while still advertising the full Content-Length. All 327 captured occurrences were clean closes.

Ruled out by measurement: network fault, crypto blocking the loop (8MB AES-GCM = 1.9ms), memory governor (zero events), HAProxy (not on this hop), pod/node specific (all 28 pods, ~25 nodes, both architectures).

Truncation lands on a page boundary — only two values ever:

190 x  received 4165632 of 8388636   (1017 x 4096)
137 x  received 8331264 of 8388636   (2034 x 4096)

Re-requesting the identical range reproduces the identical truncation (9 of 12 ranges), so retry cannot progress. Resume can — a fresh request gets a fresh response buffer.

3. Mock fidelity

async with resp["Body"] unwraps to the raw ClientResponse, whose read() takes no size argument. The shared mock returned self, keeping a size-aware read() production lacks — letting a real TypeError pass 722 tests. Reintroducing that bug now fails 24 tests.

Verification

Against the production object that fails — truncate at the observed prod offset, resume, compare:

expected len=8388636 sha=452e3406c6370759
resumed  len=8388636 sha=452e3406c6370759
IDENTICAL: True

Scope

MULTIPART_ABORTED (292) and InvalidPart are consequences — 261 of 292 aborts directly follow a fatal copy failure. Removing the 504s should remove both.

Residual risk: 4 attempts with exponential backoff drops per-copy failure to ~0.17%^4 — negligible if 504s are independent. One isolated 504 in 572 calls was observed rather than a burst, but independence under 10-node concurrent load is not proven.

fix: retry UploadPart on transient backend errors (#143)

Completes the transient-retry series for the third and last multipart operation.

RGW can accept a part, commit to a 200, stream it, and only then put the failure in the body:

InternalError: The server did not respond in time.
	status code: 200, request id: , host id:

Three retry layers all miss that shape, because each decides from the HTTP status line — which already read 200:

s3proxy is the only layer that parses the body and can see the real error.

#133 covered UploadPartCopy and #138 covered CompleteMultipartUpload. This covers plain UploadPart — the only one of the three a Scylla backup actually uses (267 PUT, 0 UploadPartCopy measured against the backup bucket).

At 50MB rclone chunks a 6.2GB SSTable is ~762 internal PUTs, so one unretried transient failure loses the whole multi-GB file.

Prod 2026-07-28: 10 of 13 racks failed, ~2.4TiB never uploaded, every failure on main.companies (47 of 48 large files).

Changes

• _upload_part_with_retry wired into both call sites in upload_part.py (_stream_and_upload_framed, _upload_internal_part_with_semaphore)
• Reuses the existing base.SOURCE_READ_ATTEMPTS machinery from #133 — no new tunables
• 8 new regression tests in tests/unit/test_upload_part_retry.py

Notes

This makes an individual PUT survive a transient upstream failure. It does not protect a PUT in flight when its pod is terminated — that trigger is KEDA scale-in cliffs, addressed separately by capping the HPA scaleDown rate.

Retry CompleteMultipartUpload on transient backend errors (#138)

Fixes prod incident 2026-07-22 where Scylla backup runs showed non-zero "Failed" bytes on some nodes. Hetzner can return 200 OK for CompleteMultipartUpload then fail mid-response with an embedded <Error>InternalError</Error> ("The server did not respond in time.") — botocore's built-in retries don't treat this as transient, so rclone failed the entire multi-GB upload instead of retrying one API call.

• Wrap complete_multipart_upload in bounded retry using existing is_retryable_source_error classifier
• On retry NoSuchUpload, recover via head_object when the object was already assembled (exact prod failure shape)
• Configurable via S3PROXY_COMPLETE_RETRY_ATTEMPTS (default 4) / S3PROXY_COMPLETE_RETRY_BACKOFF (default 0.5s)

Deploy

Bump image.tag to 2026.7.27 in argocd-system.

Hybrid passthrough for Scylla backup UploadPartCopy part 2 (#137)

Fixes InvalidPart failures on large Scylla dedup copies (e.g. main.companies *-big-Data.db) where part 2 was blocked by ranged_copy_nonzero_start and fell back to frame-by-frame streaming (~190 Hetzner GETs per 1.5GB part).

• Extend range-aware passthrough to nonzero bytes= starts (part 2 after part 1's deferred mid-frame tail)
• Add streaming_head split: consume deferred tail + re-encrypt only the misaligned frame prefix, then server-side UploadPartCopy for aligned bulk
• Remove ranged_copy_nonzero_start gate
• Add route diagnostics: hybrid_mode, passthrough segment counts, UPLOAD_PART_COPY_PART2_STREAMING_FALLBACK warning

Deploy

Bump image.tag to 2026.7.26 in argocd-system. After rollout, part 2 logs should show UPLOAD_PART_COPY_ROUTE route=passthrough hybrid_mode=head_and_passthrough, not UPLOAD_PART_COPY_STREAMING.

Graceful drain on scale-in (#136)

Chart-only release. s3proxy pods now drain instead of dying mid-upload when KEDA scales the fleet in:

• preStop sleep 10s — HAProxy stops routing new connections before the pod stops listening
• terminationGracePeriodSeconds 30 → 600 (now a values knob) — uvicorn finishes in-flight requests before exit

Fixes the remaining InvalidPart failures on Scylla backups: pods killed with 30s grace mid-UploadPart/UploadPartCopy orphaned multipart state, failing CompleteMultipartUpload (2026-07-19 23:49 UTC run error).

Fixes

• Pin uvicorn's event loop to asyncio (#135). 2026.7.22 (#134) removed the app's own uvloop.install() call, but uvicorn's default loop="auto" silently installs uvloop whenever the package is importable, so the libuv uv__io_poll abort (fd double-close under backend connection churn, upstream bug fixed-but-unreleased in MagicStack/uvloop#740) kept killing pods mid-upload. The uvicorn config now passes loop="asyncio" unless S3PROXY_UVLOOP=1 explicitly opts back in.
• The Starting log line now reports the actually-running loop class (event_loop=asyncio.SelectorEventLoop) so deployments can be verified against the runtime.

Includes #134 (2026.7.22): drop S3PROXY_MEMORY_DEBUG tracemalloc mode, gate uvloop.install() behind S3PROXY_UVLOOP.

Images: ghcr.io/serversidehannes/s3proxy-python:2026.7.24, ghcr.io/serversidehannes/s3proxy-dashboard:2026.7.24
Chart: oci://ghcr.io/serversidehannes/charts/s3proxy-python:2026.7.24

Fixes

• Retry transient source-read and segment-copy failures in UploadPartCopy (#133): a backend dropping a long-lived connection mid-body (observed against Hetzner: ClientPayloadError: received 8331264 of 8388636 bytes after ~4 min) or failing a segment copy after streaming a 200 (InternalError: The server did not respond in time) no longer fails the whole client part. Frame GETs and single-object reads retry on a fresh connection; raw copy-source streams resume with a ranged GET at the exact byte offset already delivered; passthrough segment copies retry idempotently. New env knobs: S3PROXY_SOURCE_READ_ATTEMPTS (default 4), S3PROXY_SOURCE_READ_BACKOFF (default 0.5s).

Fixes

• Listings now report plaintext size/etag for multipart objects (#131). ListObjects previously fell back to the backend (ciphertext) size for multipart-uploaded objects because their plaintext size lives in the .meta sidecar, not user metadata. Sync clients comparing sizes (scylla-manager's rclone) saw every multipart object as changed and re-uploaded it on every pass, doubling backup transfer and defeating dedup. LIST now resolves the sidecar, returns the same synthetic etag as HEAD/CompleteMultipartUpload, and caches resolved attributes in a per-pod LRU keyed by backend etag (primed at complete), so repeated listings skip the per-object round-trips.