Skip to content

Activity retry policies ignore failureDetails.isNonRetriable #317

Closed

Description

@andystaples

Summary

An activity failure marked failureDetails.isNonRetriable=true is still retried when the Python orchestration uses an explicit retry policy. The retryable-task failure path calculates the next delay without checking the flag.

I reproduced this twice with a real Python Functions host, a DTS emulator, and Azurite. The provider correctly recorded the non-retriable failure in backend history; Python then scheduled a retry timer and another activity attempt.

Related context: azure-functions-durable-extension#3582 and the follow-up on #269. This is a separate Python retry-policy issue, not a recurrence of the shared-host payload-hydration disconnect.

Environment

  • Python 3.14.10 on Windows
  • azure-functions==2.3.0
  • azure-functions-durable==2.0.0rc2
  • durabletask==1.12.0
  • Functions Core Tools 4.16.0
  • Durable Functions host extension 3.15.1
  • AzureManaged provider with host-side payloadStorageEnabled=true, payloadStorageThresholdBytes=131072, and gzip enabled
  • Python SDK-managed payload externalization disabled

Reproduction

Use an orchestration with an explicit activity retry policy and compensation handling:

@app.orchestration_trigger(context_name="context")
def orchestration(context: df.DurableOrchestrationContext):
    try:
        result = yield context.call_activity_with_retry(
            "consume_payload",
            df.RetryOptions(1000, 3),
            "x" * 160000,
        )
        return {"result": result, "compensated": False}
    except Exception:
        return {"compensated": True}


@app.activity_trigger(input_name="payload")
def consume_payload(payload: str):
    return len(payload)
  1. Gate the first activity-input blob download, then delete the committed blob before allowing the download to proceed.
  2. Use a provider that settles this missing-input delivery with a TaskFailed whose failureDetails.isNonRetriable is true. Verify that flag in raw backend history. A deterministic SDK regression test can inject the same failure event directly, without blob storage.
  3. Let the Python orchestration process that failure.

RetryOptions is the compatibility wrapper around durabletask.task.RetryPolicy; the underlying retry handling is shared.

Expected

The non-retriable failure should override the retry policy: one activity scheduling, no retry timer, and the exception should reach the orchestration's compensation handler.

Actual

Both runs recorded this sequence in raw backend history:

TaskScheduled(consume_payload)
TaskFailed(failureDetails.isNonRetriable=true)
TimerCreated
TimerFired
TaskScheduled(consume_payload)
TaskCompleted
ExecutionCompleted

There were two activity schedulings and one retry timer, despite the non-retriable flag. In this workload, the second scheduling externalized a new, readable input blob, so the retry succeeded and the orchestration returned compensated=false rather than entering its exception handler.

Without an explicit retry policy, the same provider failure reached the Python exception handler as expected. Unrelated in-flight activities remained isolated and completed normally.

Source evidence

In v1.12.0, the retryable activity failure branch calls compute_next_delay() and schedules a timer whenever a delay is available, without checking event.taskFailed.failureDetails.isNonRetriable.

The installed wheel's worker.py exactly matched the tagged source (Git blob 25010a17af0ed57143f399ece3176829387f08d3). The sub-orchestration retry branch appears to have the same omission, although I did not perform a separate live sub-orchestration retry-policy reproduction.

Suggested regression coverage

  • An activity TaskFailed with isNonRetriable=true fails the pending retryable task immediately, resumes the orchestrator, and creates neither a retry timer nor another activity attempt.
  • An otherwise identical failure with the flag unset/false retains existing retry-policy behavior.
  • Check the analogous sub-orchestration path.

This should honor an explicit non-retriable failure flag, not classify ordinary transient storage errors as permanent. Provider-side abandonment/redelivery for recoverable hydration failures is a different mechanism.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions