What is the issue?
Concurrent asynchronous RPCs can independently refresh the same expired Azure access token.
What is the impact?
When a token enters its refresh window, a burst of grpc.aio calls can trigger many credential requests at once. This adds latency, increases identity-provider load, and can contribute to credential throttling. The synchronous implementation already avoids this duplicate work.
Details about the issue including code reference
Relevant code:
|
async def get_access_token(self) -> AccessToken | None: |
|
if self._token is None or self.is_token_expired(): |
|
await self.refresh_token() |
|
return self._token |
|
|
|
def is_token_expired(self) -> bool: |
|
if self.expiry_time is None: |
|
return True |
|
return datetime.now(timezone.utc) >= ( |
|
self.expiry_time - timedelta(seconds=self._refresh_interval_seconds)) |
|
|
|
async def refresh_token(self): |
|
if self._credential is not None: |
|
self._token = await self._credential.get_token(self._scope) |
|
|
|
# Convert UNIX timestamp to timezone-aware datetime |
|
self.expiry_time = datetime.fromtimestamp(self._token.expires_on, tz=timezone.utc) |
|
self._logger.debug(f"Token refreshed. Expires at: {self.expiry_time}") |
|
async def _intercept_call( |
|
self, client_call_details: grpc.aio.ClientCallDetails) -> grpc.aio.ClientCallDetails: |
|
"""Internal intercept_call implementation which adds metadata to grpc metadata in the RPC |
|
call details.""" |
|
# Refresh the auth token if a credential was provided. The call to |
|
# get_access_token() is generally cheap, checking the expiry time and returning |
|
# the cached value without a network call when still valid. |
|
if self._token_manager is not None: |
|
access_token = await self._token_manager.get_access_token() |
|
if access_token is not None: |
|
self._upsert_authorization_header(access_token.token) |
|
|
|
return await super()._intercept_call(client_call_details) |
|
def get_access_token(self) -> AccessToken | None: |
|
if self._token is None or self.is_token_expired(): |
|
with self._refresh_lock: |
|
if self._token is None or self.is_token_expired(): |
|
self.refresh_token() |
|
return self._token |
AsyncAccessTokenManager.get_access_token() checks expiry and directly awaits refresh_token() without a lock or in-flight refresh task. DTSAsyncDefaultClientInterceptorImpl._intercept_call() awaits it for every RPC. In contrast, AccessTokenManager.get_access_token() uses double-checked locking.
A potential or proposed solution
Add an asyncio.Lock or shared in-flight refresh task. After entering the guard, re-check expiry before calling the credential. Keep the existing refresh window and ensure credential failures propagate to all waiting calls without deadlocking.
Cold-start relevance
The asynchronous client constructor correctly defers token acquisition, so this does not delay process construction. However, a concurrent burst of first RPCs while no token is cached has the same duplicate-acquisition behavior as refresh after expiry. The proposed single-flight guard therefore improves cold first-request latency under concurrency.
Relevant code:
|
def __init__(self, token_credential: AsyncTokenCredential | None, |
|
refresh_interval_seconds: int = 600): |
|
self._scope = "https://durabletask.io/.default" |
|
self._refresh_interval_seconds = refresh_interval_seconds |
|
self._logger = shared.get_logger("async_token_manager") |
|
|
|
self._credential = token_credential |
|
self._token = None |
|
self.expiry_time = None |
|
|
|
async def get_access_token(self) -> AccessToken | None: |
|
if self._token is None or self.is_token_expired(): |
|
await self.refresh_token() |
|
return self._token |
|
|
|
def is_token_expired(self) -> bool: |
|
if self.expiry_time is None: |
|
return True |
|
return datetime.now(timezone.utc) >= ( |
|
self.expiry_time - timedelta(seconds=self._refresh_interval_seconds)) |
|
|
|
async def refresh_token(self): |
|
if self._credential is not None: |
|
self._token = await self._credential.get_token(self._scope) |
|
|
|
# Convert UNIX timestamp to timezone-aware datetime |
|
self.expiry_time = datetime.fromtimestamp(self._token.expires_on, tz=timezone.utc) |
|
self._logger.debug(f"Token refreshed. Expires at: {self.expiry_time}") |
What is the issue?
Concurrent asynchronous RPCs can independently refresh the same expired Azure access token.
What is the impact?
When a token enters its refresh window, a burst of
grpc.aiocalls can trigger many credential requests at once. This adds latency, increases identity-provider load, and can contribute to credential throttling. The synchronous implementation already avoids this duplicate work.Details about the issue including code reference
Relevant code:
durabletask-python/durabletask-azuremanaged/durabletask/azuremanaged/internal/access_token_manager.py
Lines 73 to 90 in 55d8e0b
durabletask-python/durabletask-azuremanaged/durabletask/azuremanaged/internal/durabletask_grpc_interceptor.py
Lines 114 to 126 in 55d8e0b
durabletask-python/durabletask-azuremanaged/durabletask/azuremanaged/internal/access_token_manager.py
Lines 32 to 37 in 55d8e0b
AsyncAccessTokenManager.get_access_token()checks expiry and directly awaitsrefresh_token()without a lock or in-flight refresh task.DTSAsyncDefaultClientInterceptorImpl._intercept_call()awaits it for every RPC. In contrast,AccessTokenManager.get_access_token()uses double-checked locking.A potential or proposed solution
Add an
asyncio.Lockor shared in-flight refresh task. After entering the guard, re-check expiry before calling the credential. Keep the existing refresh window and ensure credential failures propagate to all waiting calls without deadlocking.Cold-start relevance
The asynchronous client constructor correctly defers token acquisition, so this does not delay process construction. However, a concurrent burst of first RPCs while no token is cached has the same duplicate-acquisition behavior as refresh after expiry. The proposed single-flight guard therefore improves cold first-request latency under concurrency.
Relevant code:
durabletask-python/durabletask-azuremanaged/durabletask/azuremanaged/internal/access_token_manager.py
Lines 63 to 90 in 55d8e0b