Repository navigation
Description
Summary
RetryPolicy.max_retry_interval does not prevent exponential-backoff overflow. RetryableTask.compute_next_delay() computes math.pow(coefficient, attempt - 1) before applying the configured cap. Once the intermediate power overflows, the worker fails the orchestration with OverflowError instead of continuing bounded retries or delivering the original exhausted failure to the orchestrator.
This reproduces with the normal coefficient 2.0, a 1ms maximum interval and 1,027 maximum attempts: the orchestration fails after 1,025 activity calls. The cap has already made the delay constant, but the unused intermediate power keeps growing. A large coefficient below accelerates the same failure to three calls.
This is a local SDK reproduction using the public in-memory testing backend, client and worker. No Azure account or service was used, and no production incident is claimed.
Reproduction
Run against core durabletask 1.13.1 from current main 256ffbdca1fb97f5ed96e547810b5d63ebefaabe with its runtime dependencies installed:
import logging
import socket
from datetime import timedelta
from durabletask import client, task, worker
from durabletask.testing import create_test_backend
logging.basicConfig(level=logging.ERROR)
with socket.socket() as s:
s.bind(('127.0.0.1',0))
port=s.getsockname()[1]
host=f'localhost:{port}'
b=create_test_backend(port=port)
calls=[]
def fail(ctx,_):
calls.append(True)
raise ValueError('downstream unavailable')
def orchestration(ctx,_):
policy=task.RetryPolicy(first_retry_interval=timedelta(milliseconds=1),max_number_of_attempts=5,
backoff_coefficient=1e200,max_retry_interval=timedelta(milliseconds=1))
try:
yield ctx.call_activity(fail,retry_policy=policy)
except task.TaskFailedError:
return 'handled expected downstream failure'
try:
with worker.TaskHubGrpcWorker(host_address=host) as w:
w.add_orchestrator(orchestration)
w.add_activity(fail)
w.start()
with client.TaskHubGrpcClient(host_address=host) as c:
id=c.schedule_new_orchestration(orchestration)
state=c.wait_for_orchestration_completion(id,timeout=10)
print('status:',state.runtime_status)
print('activity calls:',len(calls))
print('output:',state.serialized_output)
print('failure:',state.failure_details)
finally:
b.stop()
b.reset()Actual: status: FAILED, activity calls: 3, no output, and failure type builtins.OverflowError with message math range error at math.pow(...) in compute_next_delay().
Expected: five calls at delays no greater than 1ms, followed by the orchestrator's TaskFailedError handler returning handled expected downstream failure and a COMPLETED instance. With backoff_coefficient=1.0, that control does complete with five calls.
For the normal-coefficient case, change backoff_coefficient to 2.0, max_number_of_attempts to 1027 and the completion timeout to 60. I ran this separately on Python 3.13: it fails after 1,025 calls with the same SDK overflow rather than reaching the configured attempt limit. The local run completed in about 31 seconds; timing is not a benchmark or a guarantee.
Validation
Linux x86_64, CPython 3.11 and 3.13, independently:
- Unmodified native orchestration-executor, task-composition, retry-timeout and failure-details suites: 344 passed on each interpreter.
- Twelve focused delay-calculation cases: 8 fail, 4 pass as controls. Failing cases include coefficients 2, 10, 1000 and the accelerated large coefficient, with positive and zero caps.
- Four public client/worker cases: 2 fail, 2 pass as controls. Both activity and sub-orchestration retries fail for the accelerated overflow case; the constant-backoff versions exhaust normally and are caught by the parent.
- The 1,025-call coefficient-2 public activity run was executed on Python 3.13 only.
No full repository/provider suite, hosted CI, Azure emulator or live-service validation is claimed. No production fix has been written or submitted.
Scope and compatibility
I checked #96/#116 (absent maximum interval) and #316/#326 (retry-timeout logical clock); neither fixes capped-power overflow. The current open async stack retains this delay calculation too. The intended scope would be to avoid evaluating an already-saturated power while preserving attempt limits, timeout checks and ordinary delay rounding.
Correcting the failure can change replay behavior for histories that previously terminated with this overflow, so any eventual patch needs an explicit compatibility decision and upgrade guidance. I will wait for maintainer triage/discussion before preparing a PR. Would this focused capped-backoff fix be welcome?
Investigation and reproduction tests developed with AI assistance. The reproduction runs above were executed and their outputs inspected.
Activity