Repository navigation
Firestore's @transactional is not thread safe #19060
Description
Determine this is the right repository
- I determined this is the correct repository in which to report this bug.
Summary of the issue
Context
I use @firestore.transactional to run transactions in python
Expected Behavior:
I'd expect the wrapper to be thread safe and allow multiple concurrent calls since it's a method decorator.
Actual Behavior:
On retries, transactions end up with each other's retry ids, which causes failures.
Traceback (most recent call last):
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable
return callable_(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_interceptor.py", line 276, in __call__
response, ignored_call = self._with_call(
^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_interceptor.py", line 331, in _with_call
return call.result(), call
^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_channel.py", line 447, in result
raise self
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_interceptor.py", line 314, in continuation
response, call = self._thunk(new_method).with_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/grpcext/_interceptor.py", line 77, in with_call
return self._interceptor.intercept_unary(request, metadata, client_info, invoker)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/_client.py", line 177, in intercept_unary
return self._intercept(request, metadata, client_info, invoker)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/_client.py", line 158, in _intercept
raise exc
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/_client.py", line 144, in _intercept
result = invoker(request, metadata)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/grpcext/_interceptor.py", line 67, in invoker
return self._base_callable.with_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_channel.py", line 1182, in with_call
return _end_unary_response_blocking(state, call, True, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/layers/google.python.pip/pip/lib/python3.12/site-packages/grpc/_channel.py", line 999, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = "The referenced transaction has expired or is no longer valid."
debug_error_string = "INVALID_ARGUMENT:The referenced transaction has expired or is no longer valid."
API client name and version
google-cloud-firestore 2.34.1
Reproduction steps: code
file: main.py
"""Repro: concurrent calls of one @firestore.transactional function share its retry id.
Call A's commit is aborted while call B is in flight. A retries with
BeginTransaction(retry_transaction=<B's id>), which in production invalidates B, whose commit then
fails with "The referenced transaction has expired or is no longer valid".
pip install google-cloud-firestore
python transactional_race_repro.py
"""
import threading
from concurrent.futures import ThreadPoolExecutor
from google.api_core import exceptions
from google.auth.credentials import AnonymousCredentials
from google.cloud import firestore
WAIT_SECONDS = 5
class FakeRpcTransaction(firestore.Transaction):
def __init__(self, name, aborted_commits):
super().__init__(
firestore.Client(project="repro", credentials=AnonymousCredentials())
)
self.name = name
self.aborted_commits = aborted_commits
self.begun_with_retry_ids = []
def _begin(self, retry_id=None):
self.begun_with_retry_ids.append(retry_id)
self._id = f"{self.name}-{len(self.begun_with_retry_ids)}".encode()
def _commit(self):
if self.aborted_commits > 0:
self.aborted_commits -= 1
raise exceptions.Aborted("Contention")
self._clean_up()
return []
def _rollback(self):
self._clean_up()
@firestore.transactional
def update_in_transaction(transaction, began, wait_for):
# Release the GIL, as a network call in a transaction would
began.set()
assert wait_for.wait(WAIT_SECONDS)
def main():
a = FakeRpcTransaction("A", aborted_commits=1)
b = FakeRpcTransaction("B", aborted_commits=0)
a_began, b_began, a_finished = (threading.Event() for _ in range(3))
with ThreadPoolExecutor(max_workers=2) as executor:
# 1. A begins and is in flight.
a_call = executor.submit(update_in_transaction, a, a_began, b_began)
assert a_began.wait(WAIT_SECONDS)
# 2. B begins while A is in flight, resetting the shared retry id and then setting it to B's.
b_call = executor.submit(update_in_transaction, b, b_began, a_finished)
# 3. A's commit aborts, so A retries with the shared retry id, which is now B's transaction id.
a_call.result(timeout=WAIT_SECONDS)
a_finished.set()
b_call.result(timeout=WAIT_SECONDS)
print(f"google-cloud-firestore {firestore.__version__}")
print(f"A began with retry ids {a.begun_with_retry_ids}")
assert a.begun_with_retry_ids == [None, b"A-1"], (
f"A's retry began as a retry of B's transaction {a.begun_with_retry_ids[1]!r}, "
"not of its own b'A-1'"
)
if __name__ == "__main__":
main()Reproduction steps: supporting files
file: mydata.csv
alpha,1,3
beta,2,5
Reproduction steps: actual results
file: output.txtmydata.csv
Calculated: foo
Reproduction steps: expected results
file: output.txtmydata.csv
Calculated: bar
OS & version + platform
OSX
Python environment
3.12.14
Python dependencies
No response
Additional context
No response
Activity