Skip to content

Replace S3 remote signer headers instead of appending them - #4270

Merged
Fokko merged 1 commit into
apache:mainfrom
Ankitraj-builds:fix/signer-duplicate-headers
Nov 18, 2026
Merged

Fokko merged 1 commit into
apache:mainfrom
Ankitraj-builds:fix/signer-duplicate-headers

Conversation

@Ankitraj-builds

Copy link
Copy Markdown
Contributor

Closes #2422

Rationale for this change

S3V4RestSigner adds the headers returned by the signing service with add_header, which appends. A header the request already had, such as X-Amz-Content-SHA256 set by botocore, is then sent twice: once with the original value and once with the signed value. The signature only covers the signed value, so stricter S3-compatible stores reject the request. The same duplication was measured in #3896.

This deletes the existing header (case-insensitive) before adding the signed value, so each signed header is sent once, with the value the signer returned.

Are these changes tested?

Yes. test_s3v4_rest_signer_replaces_existing_headers in tests/io/test_fsspec.py sends a request that already has X-Amz-Content-SHA256 and User-Agent, and checks that after signing each header appears once with the signed value. It fails on main and passes with this change. All signer tests in the file pass.

Are there any user-facing changes?

Signed S3 requests no longer carry duplicate headers. No configuration changes.

@Fokko
Fokko added this pull request to the merge queue Nov 18, 2026
@Fokko

Fokko commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Thanks @Ankitraj-builds

Merged via the queue into apache:main with commit ce5a195 Nov 18, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

S3 remote-signer Request header case duplicated

2 participants