from pipefunc import pipefunc, Pipeline, PipeFunc
@pipefunc("y")
def f(x):
return x + 1
@pipefunc("z")
def g(y):
import random
if random.random() < 0.5:
raise ValueError("Test error.")
return y + 2
pipeline = Pipeline([f, g])
pipeline.add_mapspec_axis("x", axis="i")
r = pipeline.map({"x": range(1000)}, show_progress=True, parallel=True)
gives
It counts all unscheduled ones as failing as soon as it hits a failure.
EDIT: It turns out that it's because of chunking. If anything in the chunk fails, all fail.