Skip to content

Register the S3 signer on a session instead of a shared client - #4405

Open
Anurag-S1ngh wants to merge 2 commits into
apache:mainfrom
Anurag-S1ngh:fix/3896-s3-signer-session
Open

Anurag-S1ngh wants to merge 2 commits into
apache:mainfrom
Anurag-S1ngh:fix/3896-s3-signer-session

Conversation

@Anurag-S1ngh

Copy link
Copy Markdown
Contributor

Summary

_s3 registered the remote signer on the botocore client, which s3fs caches and every thread shares. Unregistering the signer before registering it again left a window in which a request went out without an Authorization header, and the store answered 403 against a remote-signing catalog.

Changes

  • Create an AioSession per filesystem, register the signer on its event emitter, and pass it to S3FileSystem.
  • Forward the profile property to the session, since s3fs no longer creates the session.
  • Leave the path without a signer unchanged.

This is the approach of #3783, which was closed for inactivity, with regression tests for the signer itself.

Testing

  • Added a test that the session handed to S3FileSystem signs a before-sign.s3 request, and one that two configurations get independent sessions and signers.
  • make test passes (4694 passed, 5 skipped).

Note: #4270 also mentions this issue, but it fixes the duplicate header defect from #2422 and does not change the signer registration.

Closes #3896

_s3 registered the remote signer on the botocore client, which s3fs caches
and every thread shares. Unregistering the signer before registering it
again left a window in which a request went out without an Authorization
header, and the store answered 403 against a remote-signing catalog.

Create an AioSession per filesystem, register the signer on it, and pass it
to S3FileSystem so every filesystem keeps its own signer and never touches a
shared client. The path without a signer is unchanged.
@Fokko
Fokko self-requested a review November 17, 2026 17:28

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Anurag-S1ngh! Good find, and the approach is right: the signer belongs on a session this filesystem owns, not on the shared cached client where unregister/re-register leaves an unsigned window.

I pulled the branch and verified:

  • test_s3_registers_the_signer_on_the_session and test_s3_registers_the_signer_per_session pass locally
  • Confirmed the no-signer path is untouched: _s3 passes no session kwarg when register_events is empty, so s3fs keeps creating its own session exactly as before
  • Confirmed s3fs honors an injected session: when session is passed, S3FileSystem skips creating its own (_session_is_owned = False) and uses the one we hand it, so the register_last handlers are the ones that actually fire

One nit, non-blocking: session_kwargs is typed dict[str, str] and only carries profile today. If someone later needs to forward more session-level options (e.g. a custom endpoint or credentials), the type will need loosening. Fine as is for now.

The full tests/io/test_fsspec.py run needs a local MinIO and the adlfs/gcsfs extras, so I couldn't run every S3 test locally, but CI (including integration-test-s3) is fully green. LGTM!

AioSession accepts options beyond the string profile, so keep the forwarding dict permissive.
@Anurag-S1ngh

Copy link
Copy Markdown
Contributor Author

Thanks for the review and for verifying the branch. I loosened session_kwargs to dict[str, Any] in 8dab5bf so future session options are not blocked by the annotation. Could you take another look when you have a moment?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Concurrent writes to a remote-signing catalog go out unsigned and fail 403

2 participants