Skip to content

Concurrent Loop keeps starting iterations after transmitFailed already failed the parent #19372

Description

@ayush-singh-0601

Describe the issue

LoopExecutionEventMessageHandler.handleTerminated does not check whether the parent execution / Loop task run is already terminal. After transmitFailed fails the parent, later sibling completions still start new iterations and can rewrite the Loop task FAILEDSUCCESS.

                if (loop.getTransmitFailed() && message.state().isTerminatedInError()) {
                    // ...
                    return terminateLoop(parentTaskRun, loop, executor, message.state());
                } else {
                    int nextIndex = runningIteration + terminatedIteration;
                    if (nextIndex < iterationCount) {
                        // ...
                        executionQueue.emit(loopExecution);
                    } else if (terminatedIteration == iterationCount) {
                        return terminateLoop(parentTaskRun, loop, executor, State.Type.SUCCESS);
                    }

This is not #19014 (Parallel in-process children). Loop iterations are isolated sub-executions. Tests already document that in-flight loop children keep running after the parent fails; nothing guards later LoopExecutionEvents.

Repro reasoning

  1. concurrencyLimit: 2, values: [a, b, c], transmitFailed: true (default).
  2. Iterations a and b start.
  3. a fails → terminateLoop(..., FAILED). Parent + Loop task become FAILED. b is not killed.
  4. b later succeeds. That event is not isTerminatedInError(), so the else branch runs: nextIndex = running + terminated = 2 < 3 → emits iteration c on an already-failed parent.
  5. When c is the last to finish with SUCCESS, terminatedIteration == iterationCount calls terminateLoop(..., SUCCESS). TaskRun.withState(SUCCESS) is allowed from FAILED. The Loop task is rewritten SUCCESS while the execution can stay FAILED.

loopExecution() also copies this.state from the parent, so post-fail children can be born already FAILED.

Fix: at the top of handleTerminated, if execution.getState().isTerminated() or parentTaskRun.getState().isTerminated(), only bookkeep outputs (optional) and return. Never emit a new iteration or call terminateLoop again. On transmitFailed, kill remaining loop sub-executions (same idea as Parallel onChildFailure).

Environment

  • Kestra Version: develop (37e4d38)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/backendNeeds backend code changes

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions