Skip to content

If anything in chunk fails, the entire chunk is marked as failed. #765

@basnijholt

Description

@basnijholt
from pipefunc import pipefunc, Pipeline, PipeFunc


@pipefunc("y")
def f(x):
    return x + 1


@pipefunc("z")
def g(y):
    import random

    if random.random() < 0.5:
        raise ValueError("Test error.")
    return y + 2


pipeline = Pipeline([f, g])
pipeline.add_mapspec_axis("x", axis="i")

r = pipeline.map({"x": range(1000)}, show_progress=True, parallel=True)

gives

Image

It counts all unscheduled ones as failing as soon as it hits a failure.

EDIT: It turns out that it's because of chunking. If anything in the chunk fails, all fail.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions