Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: ServerSideHannes/s3proxy-python
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: main
Choose a base ref
...
head repository: ServerSideHannes/s3proxy-python
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: feat/frame-streaming-bodies
Choose a head ref
Checking mergeability… Don’t worry, you can still create the pull request.
  • 3 commits
  • 19 files changed
  • 1 contributor

Commits on Jul 10, 2026

  1. feat: disable backend payload signing, verify ciphertext MD5 against … 

    …backend ETag

Backend SigV4 payload signing forced botocore to hold a second full copy of
every uploaded internal part. Switch the backend hop to UNSIGNED-PAYLOAD and
flexible checksums to when_required, and replace the lost integrity check with
an incremental ciphertext-MD5 vs backend-ETag comparison on every upload_part
and put_object (BackendIntegrityError -> 500 InternalError -> client retries).
Non-MD5 ETags (SSE-KMS backends) skip verification with a one-time warning.

Closes #127
    @ServerSideHannes
    ServerSideHannes committed Jul 10, 2026
    Configuration menu
    Copy the full SHA
    785dc0f View commit details
    Browse the repository at this point in the history

  2. feat: stream sealed frames as upload bodies — O(part) -> O(frame) memory 

    Upload paths buffered a full internal part of ciphertext before handing it to
the backend (up to ~240MB for multi-GB copies; the reason pods need 1Gi).
FramedStreamBody now yields sealed 8MiB AES-GCM frames (in 1MB slices) as
encryption produces them, with exact Content-Length via framed_ciphertext_size
and a running ciphertext MD5 for backend ETag verification.

- UploadPart framed path: multi-frame internal parts stream; single-frame
  parts stay buffered bytes so botocore can replay them. A mid-part backend
  failure surfaces as a failed client part (rclone/barman re-send it).
- Copy pump: internal parts stream with pump-owned retry — on failure the
  plaintext source is reopened at the part's offset (ranged re-read) and the
  client-part MD5 is committed per attempt from a copy() snapshot, so retried
  bytes are never double-hashed. Short sources now fail loudly instead of
  writing wrong sizes.
- Governor formulas recalibrated (tracemalloc): streaming_upload_peak flat
  4*FRAME (32MB, real ~24MB) for multi-frame parts; copy_chunk_peak flat
  4*FRAME + 2*MAX_BUFFER (48MB, real ~32MB). Clamp machinery is now vestigial.

Closes #128
    @ServerSideHannes
    ServerSideHannes committed Jul 10, 2026
    Configuration menu
    Copy the full SHA
    d417cb4 View commit details
    Browse the repository at this point in the history

  3. fix: clamp copy expected bytes to real segment extent; recalibrate re… 

    …maining governor tests

Scylla manifest sidecars overstate total_plaintext_size relative to the stored
segments; the pump's strict Content-Length expectation must come from the
segments' real extent or every inflated-metadata copy aborts as a short source.
Also update the four remaining tests pinned to the old peak formulas.
    @ServerSideHannes
    ServerSideHannes committed Jul 10, 2026
    Configuration menu
    Copy the full SHA
    13bdf59 View commit details
    Browse the repository at this point in the history
Loading