Repository navigation
Description
Apache Iceberg version
0.12.0 (latest release). The same code is on main.
Please describe the bug 🐞
With a request signer configured (s3.signer, e.g. REST remote signing), FsspecFileIO intermittently sends unsigned S3 requests. The store answers 403, which s3fs turns into PermissionError: Forbidden. It is usually a HeadObject, because that is the most frequent call.
We saw this in long-running write jobs against a REST catalog (Lakekeeper) with remote signing on an S3-compatible store. The 403s were rare and looked random, and a retry always succeeded.
Cause
pyiceberg.io.fsspec._s3 does the following:
fs = S3FileSystem(**s3_fs_kwargs)
for event_name, event_function in register_events.items():
fs.s3.meta.events.unregister(event_name, unique_id=1925)
fs.s3.meta.events.register_last(event_name, event_function, unique_id=1925)- The S3FileSystem is shared.
S3FileSystem(**s3_fs_kwargs)comes from fsspec's instance cache. The signer is not part of the cache key, so everyFsspecFileIOin the process with the same endpoint/region gets the sameS3FileSystemand the same aiobotocore client. - The signer is swapped while requests are in flight. Every call to
_s3unregisters the signer on that shared client and registers its own.FsspecFileIO.get_fscaches per thread, so_s3runs once per FileIO per thread, i.e. every time a newly loaded table is read or written from a worker thread. - The gap sends unsigned requests. fsspec's IO thread keeps signing other requests during the swap. A
before-sign.s3emitted betweenunregisterandregister_lastfinds no signer, and sincesignature_versionisUNSIGNED, the request goes out unsigned. Occasionally the two steps also race each other and raiseKeyError: 1925from botocore'sunregister.
A second consequence, without any race: all FileIOs sign with the signer of whichever FileIO called _s3 last. That signer carries that FileIO's properties: the signer URI/endpoint of that table and the AuthManager of that catalog. A process with two catalogs (different principals or warehouses) can therefore sign one catalog's requests with the other's credentials.
A related, smaller issue: the signer is registered on the client object. If s3fs ever recreates its client (set_session does so when the aiohttp sessions are closed), the new client has no signer.
Reproduction
The script below needs only MinIO, or any S3 that rejects unsigned requests:
docker run -p 9000:9000 -e MINIO_ROOT_USER=minio -e MINIO_ROOT_PASSWORD=minio123 minio/minio server /data
It registers an in-process signer in SIGNERS that signs with SigV4, as a REST signer would. Four threads HEAD one object while two threads create new FsspecFileIOs and call get_fs("s3"), which is what loading a table and reading it from a worker does.
repro.py
"""Shared S3FileSystem + per-FileIO signer swap in pyiceberg.io.fsspec._s3.
Needs a MinIO server (or any S3 that rejects unsigned requests), e.g.:
docker run -p 9000:9000 -e MINIO_ROOT_USER=minio -e MINIO_ROOT_PASSWORD=minio123 minio/minio server /data
"""
import collections
import sys
import threading
import time
import s3fs
from botocore.auth import S3SigV4Auth
from botocore.credentials import Credentials
from pyiceberg.io import fsspec as pf
ENDPOINT = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:9000"
KEY, SECRET = (sys.argv[2], sys.argv[3]) if len(sys.argv) > 3 else ("minio", "minio123")
class InProcessSigner(pf.S3RequestSigner):
"""Stands in for S3V4RestSigner: signs the request the way a REST signer would."""
def __call__(self, request, **_):
S3SigV4Auth(Credentials(KEY, SECRET), "s3", "us-east-1").add_auth(request)
pf.SIGNERS["InProcessSigner"] = InProcessSigner
PROPS = {"s3.endpoint": ENDPOINT, "s3.region": "us-east-1", "s3.signer": "InProcessSigner"}
admin = s3fs.S3FileSystem(key=KEY, secret=SECRET, skip_instance_cache=True,
client_kwargs={"endpoint_url": ENDPOINT, "region_name": "us-east-1"})
if not admin.exists("repro"):
admin.mkdir("repro")
admin.pipe("repro/data.parquet", b"x" * 100)
# 1. Every FsspecFileIO with the same endpoint/region gets the same S3FileSystem.
a, b = pf.FsspecFileIO({**PROPS, "table": "a"}), pf.FsspecFileIO({**PROPS, "table": "b"})
print("same S3FileSystem for two FileIOs:", a.get_fs("s3") is b.get_fs("s3"))
def run(seconds, new_fileios):
fs = pf.FsspecFileIO(dict(PROPS)).get_fs("s3")
ok, errors, stop = [0], collections.Counter(), threading.Event()
def reader():
while not stop.is_set():
try:
fs.info("repro/data.parquet", refresh=True)
ok[0] += 1
except Exception as e: # noqa: BLE001
errors[f"{type(e).__name__}: {e}"[:60]] += 1
def loader():
# What happens whenever a table is loaded and then read/written from a worker thread.
while not stop.is_set():
try:
pf.FsspecFileIO(dict(PROPS)).get_fs("s3")
except Exception as e: # noqa: BLE001
errors[f"loader {type(e).__name__}: {e}"[:60]] += 1
threads = [threading.Thread(target=reader) for _ in range(4)]
threads += [threading.Thread(target=loader) for _ in range(new_fileios)]
for t in threads:
t.start()
time.sleep(seconds)
stop.set()
for t in threads:
t.join()
return ok[0], dict(errors)
# 2. Unsigned requests while other threads create FileIOs.
print("readers only: ", run(20, 0))
print("readers + 2 FileIO loops:", run(20, 2))Output (pyiceberg 0.12.0, s3fs/fsspec 2026.9.0, botocore 1.43.117, aiobotocore 3.10.0, Python 3.13, macOS), two runs:
same S3FileSystem for two FileIOs: True
readers only: (44788, {})
readers + 2 FileIO loops: (486, {'PermissionError: Forbidden': 27, 'loader KeyError: 1925': 4})
readers only: (43351, {})
readers + 2 FileIO loops: (493, {'PermissionError: Forbidden': 8, 'loader KeyError: 1925': 7})
The loaders run in a tight loop to make the race frequent. In real workloads the rate is much lower, which is why it showed up as rare 403s in long jobs.
Possible fix
When a signer is configured, don't put per-FileIO state on a process-wide cached client.
- Create the filesystem outside the instance cache:
S3FileSystem(**s3_fs_kwargs, skip_instance_cache=True). - Cache it on the FileIO, shared by all its threads, instead of once per thread.
- Register the signer once, before the filesystem is handed out, and never unregister it.
Alternatively, include the signer identity in the cache key. That fixes the race but not the "last signer wins" problem across catalogs.
To also cover client recreation, register the signer in an S3FileSystem.set_session override, or check it there.
We currently work around it in our own FsspecFileIO subclass by doing exactly that: an own S3FileSystem subclass per FileIO, with skip_instance_cache=True and the signer registered in set_session. With that, the repro above shows 0 errors.
Willingness to contribute
- I can contribute a fix for this bug independently
- I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- I cannot contribute a fix for this bug at this time
Activity
SurenderguliaEnginner commentedon Nov 18, 2026
Hi! I'd like to work on a fix and submit a PR for this if you aren't already working on it.
workbench13 commentedon Nov 18, 2026
Feel free, running ourselves on the fix outlined in the report.