Skip to content

Add deterministic async/await orchestrators across all three Python packages - #293

Closed
andystaples wants to merge 22 commits into
mainfrom
andystaples-async-orchestrator-spike
Closed

andystaples wants to merge 22 commits into
mainfrom
andystaples-async-orchestrator-spike

Conversation

@andystaples

@andystaples andystaples commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds coroutine-first orchestration to durabletask, durabletask.azuremanaged, and azure-functions-durable. An async def orchestrator automatically receives an async context; no conversion call is needed. Existing generator orchestrators retain their eager durable task handles, and both styles use the same history and replay engine. Durable parents and children can use either style.

from durabletask import task


async def greet(ctx: task.AsyncOrchestrationContext, name: str) -> str:
    return await ctx.call_activity("say_hello", input=name, return_type=str)

Functions also supports single-argument async orchestrators with df.AsyncDurableOrchestrationContext, preserving the v1-style method and argument names, including durable HTTP retry options.

Behavior and limitations

  • Async context operations are cold, single-use native coroutines. Validation, serialization, action allocation, and relative timer deadlines begin when a coroutine starts. Generator-context operations remain eager. as_task_context() and Task.wait() provide explicit interoperability with reusable durable handles.
  • The documented deterministic asyncio subset includes ordered gather, task groups, local tasks and futures, durable sleeps, and timeouts. asyncio.wait, as_completed, arbitrary third-party async libraries, external I/O, threads, and subprocesses are outside the contract. Nondeterministic work belongs in activities.
  • The Functions compatibility context provides async task_all and task_any; grouping preserves input order and duplicate references, and supplied futures are borrowed rather than implicitly cancelled.
  • Cancelling an owned event wait or timer withdraws its local registration, but cannot undo already-issued remote work or side effects. A cancelled pending entity-lock acquisition may still be granted; retain and shield the acquisition if it must later be released.
  • Dispatch-local tasks and scopes are cleaned up at every dispatch boundary, including suspension. Finalizers may run before logical orchestration completion and cannot schedule or await new durable work. Durable compensation belongs in normal orchestration control flow.

Replay and release considerations

Converting an in-flight generator orchestration requires preserving replay-visible actions, ordering, payloads, timers, event consumption, and entity-release behavior. Python/asyncio upgrades can also change scheduling; validate recorded histories before changing runtimes for in-flight instances. See the async orchestration contract for the supported APIs and upgrade guidance.

All three package changelogs document the feature under Unreleased. Provider releases require the corresponding core capability; package versions and dependency minimums should be updated together in a coordinated release PR after the core release.

Relates to #294.

Make durable tasks awaitable, drive native coroutines through the existing replay engine, and preserve generator compatibility. Include provider smoke coverage, prototype documentation, and coordinated-release notes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@andystaples andystaples linked an issue Oct 15, 2026 that may be closed by this pull request
5 tasks
Tamir Dresher (tamirdresher) added a commit that referenced this pull request Oct 20, 2026
Give each replay dispatch explicit ownership of its orchestrator generator: close it once in an outer finally after the response is built, so cleanup in finally blocks runs deterministically and cannot alter the returned actions or custom status. Ordinary teardown failures are logged without replacing dispatch results or abandonment.

Async bridges such as Pydantic AI otherwise retain AnyIO task scopes, event loops, and orchestration contexts indefinitely. Also relevant to the async/await work in #293 and #294.

Fixes #299

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35371373-86cf-4897-bf7c-460fa3c88d1f
andystaples and others added 5 commits October 30, 2026 12:21
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve generator dispatch recovery and snapshots; resume coroutine branches per history completion, order cancellation wakeups, and define the bounded async contract.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Handle closure tasks created during cleanup, retaining warnings for ordinary errors and terminal recovery for unexpected BaseException exits.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Exercise TestModel parallel tools and recorded failures without a trampoline, retain adapter serialization and replay-stable metadata, and cover Windows Python versions in CI.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
andystaples and others added 2 commits October 30, 2026 12:50
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@andystaples andystaples changed the title Prototype async/await orchestrators across all three Python packages Add deterministic async/await orchestrators across all three Python packages Oct 30, 2026
Continue bounded cleanup after callback and process-control failures. Deliver durable completions through ordered subscriptions instead of repeated pending-work scans, and scale the cleanup budget for large cooperative fan-out. Add focused regressions and keep documentation adoption guidance minimal.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
andystaples and others added 2 commits November 2, 2026 10:19
Preserve the async completion observer and cleanup fixes while adopting main's ExceptionGroup control-flow classification and Python 3.11 minimum. Resolve the provider changelog by retaining both changes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Callback-originated StopIteration can silently complete a still-pending workflow, while duplicate timer registrations can overwrite observers and cleanup failures can mask process-control exceptions.

3 open issues
What changed in this PR

Adds native async/await orchestrators across all three Python packages using the shared replay engine, while retaining generator support and mixed-style sub-orchestrations.

Changes:

  • Adds deterministic asyncio scheduling, durable timers, local cancellation, and dispatch cleanup.
  • Adds core and provider tests, Functions host smoke tests, a Pydantic AI integration proof, and Windows coverage.
  • Documents supported async APIs, replay constraints, and provider release coordination.
File Description
tests/​durabletask/​test_pydantic_ai_orchestrator.py Tests framework model/tool activities across replay.
tests/​durabletask/​test_coroutine_orchestrators.py Covers direct awaits and existing durable operations.
tests/​durabletask/​test_asyncio_orchestrators.py Tests scheduling, timers, cancellation, and cleanup.
tests/​durabletask-azuremanaged/​test_dts_coroutine_orchestrators.py Tests mixed orchestrators over core and managed transports.
tests/​azure-functions-durable/​test_functions_coroutine_orchestrators.py Tests Functions entry points and coroutine mocks.
tests/​azure-functions-durable/​e2e/​test_coroutine_e2e.py Adds real-host async smoke coverage.
tests/​azure-functions-durable/​e2e/​apps/​v1_style/​orchestrators.py Adds a compatibility-style async probe.
tests/​azure-functions-durable/​e2e/​apps/​dtask_style/​orchestrators.py Adds a native-style async probe.
requirements.txt Adds the framework-proof dependency.
README.md Introduces async authoring and its boundaries.
durabletask/​worker.py Integrates coroutine execution and teardown.
durabletask/​task.py Makes durable tasks awaitable and extends orchestrator typing.
durabletask/​internal/​asyncio_loop.py Implements the dispatch-local deterministic event loop.
durabletask-azuremanaged/​CHANGELOG.md Records managed-provider async support.
docs/​supported-patterns.md Links async patterns and clarifies cancellation behavior.
docs/​async-orchestrators.md Defines async scheduling, cleanup, and replay contracts.
CHANGELOG.md Records core async capabilities.
azure-functions-durable/​README.md Describes both Functions async authoring styles.
azure-functions-durable/​CHANGELOG.md Records Functions async and testing support.
azure-functions-durable/​azure/​durable_functions/​testing/​orchestrator_generator_wrapper.py Supports direct-await coroutine unit tests.
.github/​workflows/​durabletask.yml Adds Windows async-runtime test coverage.

Review effort: Balanced

Comment thread durabletask/internal/asyncio_loop.py Resolved
Comment thread durabletask/internal/asyncio_loop.py Resolved
Comment thread durabletask/internal/asyncio_loop.py Resolved
Keep identical timer registrations independent and reject callback StopIteration as a failure. Replace the custom Pydantic integration proof with AnyIO context/replay coverage and preserve cleanup control-flow precedence.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The wrapper fails on direct durable awaits in async tests with a running event loop, while completed-task retention keeps discarded results alive and causes memory usage to grow across replay batches.

No open issues 🎉

2 previously missed issues

In code that hasn't changed since last review

Medium severity Durable awaits fail when called from an async test loop

azure-functions-durable/​azure/​durable_functions/​testing/​orchestrator_generator_wrapper.py:42

Calling this wrapper from an async test with a running asyncio loop fails even for a direct durable await: Task.__await__ sees the test's loop and raises "Durable tasks can only be awaited inside a durable orchestrator." Isolate every coroutine send/throw from the ambient loop and restore the caller's state afterward. Add an async-test regression for direct durable awaits.

Medium severity Completed tasks retain results for the entire replay

durabletask/​internal/​asyncio_loop.py:132

_tasks retains every completed task until dispatch teardown, including each successful task's result. Replaying sequential, discarded gather batches therefore keeps earlier decoded activity results alive, so memory grows with the full replay rather than the live batch. Retire successful completed tasks and use a separate monotonic creation counter for cancellation ordering, while preserving tracking for pending tasks and unobserved failures.

3 resolved since last review

Review effort: Balanced

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Comment thread durabletask/worker.py Fixed
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
andystaples and others added 3 commits November 3, 2026 10:18
Retain released durabletask 1.13.0 and Functions 2.0.0rc4 metadata. Cover non-retriable failures through coroutine replay.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Replace AnyIO test usage with asyncio task groups and retain replay, context propagation, cancellation ordering, and cleanup coverage. Remove the test dependency and third-party compatibility claims.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Retire completed local tasks when their failures are observed, including after the initial completion callback. Preserve pending-task tracking and unobserved background failure reporting.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Expose an explicit native-coroutine facade while preserving eager durable handles and existing generator APIs. Clean up owned event registrations and timers without changing borrowed-operation cancellation semantics.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Comment thread durabletask/async_context.py Resolved
Comment thread durabletask/async_context.py Resolved
Comment thread durabletask/async_context.py Resolved
Comment thread durabletask/async_context.py Resolved
Comment thread durabletask/async_context.py Resolved
Comment thread durabletask/async_context.py Resolved
Comment thread durabletask/async_context.py Resolved
Comment thread durabletask/async_context.py Resolved
Comment thread durabletask/async_context.py Fixed
Comment thread durabletask/async_context.py Fixed

@berndverst Bernd Verst (berndverst) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the shared async replay implementation and the core, Azure Managed and Functions integration paths. I found no blocking correctness or compatibility defects at 0872493. The additive API preserves existing generator authoring and avoids introducing new abstract requirements; replay parity and provider coverage are substantial.

The main tradeoff is replay CPU: a local synthetic sequential replay benchmark was approximately 2.2x slower for equivalent async code than generators, while an executor-only comparison showed no material slowdown for existing generators. This is opt-in overhead, not a merge blocker, but is relevant for long histories.

The documented restrictions on external I/O, dispatch-local cleanup, migration of persisted histories and Python-version-dependent scheduling are important. Core regressions passed locally on 3.11/3.13/3.14, provider tests passed, and the live CI matrix is green. No changes requested.

andystaples and others added 3 commits November 10, 2026 11:35
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Supply coroutine-native contexts automatically while preserving generator orchestration and eager-handle interoperability across providers.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Comment thread durabletask/async_context.py Resolved
Comment thread durabletask/async_context.py Resolved
@andystaples

Copy link
Copy Markdown
Contributor Author

Superseded by the review stack #356 → #357 → #358 → #360 → #361 → #363 → #364. The top of the stack (#364) has a tree identical to this PR's final state. Review each layer independently; the stack is squash-merged as a whole once every layer is approved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support async/await orchestrators across the Python SDK packages

3 participants