Repository navigation
Register the S3 signer on a session instead of a shared client - #4405
Anurag-S1ngh wants to merge 2 commits into
Conversation
_s3 registered the remote signer on the botocore client, which s3fs caches and every thread shares. Unregistering the signer before registering it again left a window in which a request went out without an Authorization header, and the store answered 403 against a remote-signing catalog. Create an AioSession per filesystem, register the signer on it, and pass it to S3FileSystem so every filesystem keeps its own signer and never touches a shared client. The path without a signer is unchanged.
Fokko
left a comment
There was a problem hiding this comment.
Thanks @Anurag-S1ngh! Good find, and the approach is right: the signer belongs on a session this filesystem owns, not on the shared cached client where unregister/re-register leaves an unsigned window.
I pulled the branch and verified:
test_s3_registers_the_signer_on_the_sessionandtest_s3_registers_the_signer_per_sessionpass locally- Confirmed the no-signer path is untouched:
_s3passes nosessionkwarg whenregister_eventsis empty, so s3fs keeps creating its own session exactly as before - Confirmed s3fs honors an injected session: when
sessionis passed,S3FileSystemskips creating its own (_session_is_owned = False) and uses the one we hand it, so theregister_lasthandlers are the ones that actually fire
One nit, non-blocking: session_kwargs is typed dict[str, str] and only carries profile today. If someone later needs to forward more session-level options (e.g. a custom endpoint or credentials), the type will need loosening. Fine as is for now.
The full tests/io/test_fsspec.py run needs a local MinIO and the adlfs/gcsfs extras, so I couldn't run every S3 test locally, but CI (including integration-test-s3) is fully green. LGTM!
AioSession accepts options beyond the string profile, so keep the forwarding dict permissive.
|
Thanks for the review and for verifying the branch. I loosened |
Summary
_s3registered the remote signer on the botocore client, which s3fs caches and every thread shares. Unregistering the signer before registering it again left a window in which a request went out without anAuthorizationheader, and the store answered 403 against a remote-signing catalog.Changes
AioSessionper filesystem, register the signer on its event emitter, and pass it toS3FileSystem.profileproperty to the session, since s3fs no longer creates the session.This is the approach of #3783, which was closed for inactivity, with regression tests for the signer itself.
Testing
S3FileSystemsigns abefore-sign.s3request, and one that two configurations get independent sessions and signers.make testpasses (4694 passed, 5 skipped).Note: #4270 also mentions this issue, but it fixes the duplicate header defect from #2422 and does not change the signer registration.
Closes #3896