Skip to content

Commit d0e50e2

Browse files
timsaucerclaude
andcommitted
Number the stages, and ship all of them
`insert_stage` wrapped every topmost partial aggregate but gave each one the same `STAGE_ID`. Two stages sharing an id exchange results through the same `stage-1-part-N.arrow` paths, and a union drives its branches concurrently, so they race. A `UNION ALL` of two `GROUP BY`s fails outright: Internal error: dfx_engine: publishing .../stage-1-part-1.arrow: No such file or directory (os error 2) Three defects, one symptom: - Stage ids are now allocated from a counter threaded through `insert_stage`, not a constant. Threaded rather than global because the driver plans the same query twice -- once to ship the stages, once to read their results back -- and only per-call numbering has both calls agree on which stage is which. `stages_inserted` counts every stage instead of once per planning call. - The temporary file a writer builds a partition in is unique to that writer rather than derived from the final name. Two writers of one partition were interleaving their batches into a single file and then one rename found it already consumed, which is why the failure surfaced as a rename error. The rename that is supposed to make publishing atomic was the thing that broke. - The driver ships every stage. `find_stages` returns all of them in pre-order and `run_distributed` dispatches one worker per `(stage_id, partition)`; leaving one behind had the driver compute it locally while the plan it shipped claimed otherwise. `DistributedResult.partitions`/`worker_rows` become `tasks`/`task_rows` keyed by both, since partition 0 of two stages is two pieces of work. Python cannot read a stage id back off a plan -- the FFI wrapper's display replaces `ShuffleStageExec: stage=2` -- so the numbering is a convention shared with Rust, and `_internal.stage_id(index)` is the one place the arithmetic is written down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent fc14b14 commit d0e50e2

7 files changed

Lines changed: 245 additions & 92 deletions

File tree

‎examples/distributed/README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,9 @@ It holds one partition of results in memory before writing, because an Arrow
129129
IPC stream needs its schema up front. A production engine would stream to the
130130
file and track completion separately.
131131

132-
It has one stage. A real engine chains them, and the interesting problems —
132+
It has one *level* of stages. A query with an aggregate in each branch of a
133+
union gets one stage per branch, and they run side by side — but no stage ever
134+
feeds another. A real engine chains them, and the interesting problems —
133135
scheduling, retries, straggler handling, memory limits — all live in the part
134136
this example replaces with `subprocess.Popen` and a `for` loop.
135137

‎examples/distributed/engine-library/python/dfx_engine/driver.py‎

Lines changed: 84 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -18,10 +18,10 @@
1818
"""The driver: split a query into tasks, fan them out, collect the answer.
1919
2020
The shape is deliberately boring, because the interesting part is not the
21-
scheduling. What matters is the four things the driver has to get right, each
21+
scheduling. What matters is the five things the driver has to get right, each
2222
of which is a way a real deployment goes wrong:
2323
24-
1. It serializes the stage **with** its session. ``to_bytes(None)`` uses an
24+
1. It serializes each stage **with** its session. ``to_bytes(None)`` uses an
2525
empty codec chain and cannot encode any library's node.
2626
2. It ships the codec ids it used, so a worker can refuse a plan it would
2727
misread rather than decode it with the wrong codec.
@@ -30,6 +30,9 @@
3030
cannot disagree.
3131
4. It waits for every worker before reading, because the stage node decides
3232
whether to read or recompute by looking at the filesystem.
33+
5. It ships *every* stage. A plan can hold more than one -- an aggregate in
34+
each branch of a union, say -- and they are independent subtrees rather
35+
than a chain.
3336
"""
3437

3538
from __future__ import annotations
@@ -49,7 +52,12 @@
4952
from datafusion.plan import ExecutionPlan
5053
from datafusion.user_defined import ScalarUDF
5154

52-
__all__ = ["DistributedResult", "find_stage", "run_distributed"]
55+
__all__ = [
56+
"DistributedResult",
57+
"find_stage",
58+
"find_stages",
59+
"run_distributed",
60+
]
5361

5462

5563
class DistributedResult:
@@ -58,21 +66,26 @@ class DistributedResult:
5866
def __init__(
5967
self,
6068
batches: list[pa.RecordBatch],
61-
partitions: list[int],
62-
worker_rows: dict[int, int],
69+
tasks: list[tuple[int, int]],
70+
task_rows: dict[tuple[int, int], int],
6371
) -> None:
6472
self.batches = batches
65-
self.partitions = partitions
66-
"""Partition indices that were dispatched, one per worker."""
67-
self.worker_rows = worker_rows
68-
"""Rows each worker produced, keyed by partition index."""
73+
self.tasks = tasks
74+
"""``(stage_id, partition)`` pairs that were dispatched, one per worker.
75+
76+
Keyed by both, not by partition alone: a query with an aggregate in
77+
more than one branch has more than one stage, and partition 0 of each
78+
is a different piece of work.
79+
"""
80+
self.task_rows = task_rows
81+
"""Rows each worker produced, keyed as :attr:`tasks` is."""
6982

7083

7184
STAGE_NODE_NAME = "ShuffleStageExec"
7285

7386

74-
def find_stage(plan: ExecutionPlan) -> ExecutionPlan | None:
75-
"""Locate the stage node the planner inserted.
87+
def find_stages(plan: ExecutionPlan) -> list[ExecutionPlan]:
88+
"""Locate every stage node the planner inserted, in pre-order.
7689
7790
Matched on the display string because a Python caller has no way to
7891
downcast a Rust plan node -- there is no ``isinstance`` across an FFI
@@ -87,27 +100,39 @@ def find_stage(plan: ExecutionPlan) -> ExecutionPlan | None:
87100
A foreign node reports its own name nested inside the wrapper's, which
88101
makes anchored matches on plan text quietly wrong -- the kind of thing
89102
that works in a single-library test and fails the moment a real extension
90-
is involved.
103+
is involved. It is also why the stage *id* has to be recomputed here
104+
rather than read: the wrapper dropped it.
105+
106+
Pre-order is the contract with the planner, which numbers stages in the
107+
same walk, so the nth node returned here has the id
108+
``_internal.stage_id(n)``. The recursion stops at a stage rather than
109+
descending into it, because the planner never nests one inside another.
91110
"""
92111
if STAGE_NODE_NAME in plan.display():
93-
return plan
94-
for child in plan.children():
95-
found = find_stage(child)
96-
if found is not None:
97-
return found
98-
return None
112+
return [plan]
113+
return [stage for child in plan.children() for stage in find_stages(child)]
114+
115+
116+
def find_stage(plan: ExecutionPlan) -> ExecutionPlan | None:
117+
"""The first stage in `plan`, or ``None``.
118+
119+
For tests and callers that only care whether the planner split at all.
120+
See :func:`find_stages`.
121+
"""
122+
stages = find_stages(plan)
123+
return stages[0] if stages else None
99124

100125

101126
def _dispatch(
102-
envelope: dict, envelope_dir: pathlib.Path, partition: int
127+
envelope: dict, envelope_dir: pathlib.Path, stage_id: int, partition: int
103128
) -> subprocess.Popen[str]:
104-
"""Start one worker for one partition.
129+
"""Start one worker for one ``(stage, partition)``.
105130
106131
``sys.executable``, not ``python``: a worker on a different Python minor
107132
version cannot load a cloudpickled inline UDF, and that failure is far
108133
from its cause.
109134
"""
110-
path = envelope_dir / f"task-{partition}.json"
135+
path = envelope_dir / f"task-{stage_id}-{partition}.json"
111136
path.write_text(json.dumps(envelope))
112137
return subprocess.Popen( # noqa: S603
113138
[sys.executable, "-m", "dfx_engine.worker", str(path)],
@@ -120,7 +145,7 @@ def _dispatch(
120145
def run_distributed(
121146
sql: str, spec: SessionSpec, extra_udfs: list[ScalarUDF] | None = None
122147
) -> DistributedResult:
123-
"""Run `sql`, executing its leaf stage in one worker process per partition.
148+
"""Run `sql`, executing each stage partition in its own worker process.
124149
125150
Requires ``spec.shuffle_dir``: without it the planner inserts no stage and
126151
there is nothing to distribute.
@@ -135,13 +160,13 @@ def run_distributed(
135160
message = "run_distributed needs a shuffle_dir; build_session got none"
136161
raise ValueError(message)
137162

138-
ctx, engine, _storage = build_session(spec)
163+
ctx, _engine, _storage = build_session(spec)
139164
for function in extra_udfs or []:
140165
ctx.register_udf(function)
141166
plan = ctx.sql(sql).execution_plan()
142167

143-
stage = find_stage(plan)
144-
if stage is None:
168+
stages = find_stages(plan)
169+
if not stages:
145170
message = (
146171
"no ShuffleStageExec in the plan; the engine's planner did not run, "
147172
"or its config extension was not registered"
@@ -151,48 +176,56 @@ def run_distributed(
151176
shuffle_dir = pathlib.Path(spec.shuffle_dir)
152177
shuffle_dir.mkdir(parents=True, exist_ok=True)
153178

154-
# Encode the stage subtree, through the session that owns the codecs.
155-
plan_path = shuffle_dir / "stage.plan"
156-
plan_path.write_bytes(stage.to_bytes(ctx))
157-
158-
stage_id = _internal.stage_id()
159-
partitions = list(range(stage.partition_count))
160-
envelopes = [
161-
{
162-
"spec": spec.to_json(),
163-
"plan": str(plan_path),
164-
"stage_id": stage_id,
165-
"partition": partition,
166-
}
167-
for partition in partitions
168-
]
169-
170-
# One process per partition, all in flight together. This is the claim the
179+
# One task per (stage, partition). Every stage is shipped, not just the
180+
# first: a query with an aggregate in two branches has two independent
181+
# subtrees, and leaving one behind would have the driver compute it
182+
# locally while the plan it shipped claimed otherwise.
183+
tasks: list[tuple[int, int]] = []
184+
envelopes = []
185+
for index, stage in enumerate(stages):
186+
# The id the planner gave this stage, recomputed from its position
187+
# because the FFI wrapper's display does not carry it.
188+
stage_id = _internal.stage_id(index)
189+
# Encode the stage subtree, through the session that owns the codecs.
190+
plan_path = shuffle_dir / f"stage-{stage_id}.plan"
191+
plan_path.write_bytes(stage.to_bytes(ctx))
192+
for partition in range(stage.partition_count):
193+
tasks.append((stage_id, partition))
194+
envelopes.append(
195+
{
196+
"spec": spec.to_json(),
197+
"plan": str(plan_path),
198+
"stage_id": stage_id,
199+
"partition": partition,
200+
}
201+
)
202+
203+
# One process per task, all in flight together. This is the claim the
171204
# example is making: each worker reads a different file and writes a
172205
# different result, so they need no coordination beyond the directory.
173206
workers = [
174-
_dispatch(envelope, shuffle_dir, partition)
175-
for envelope, partition in zip(envelopes, partitions, strict=True)
207+
_dispatch(envelope, shuffle_dir, stage_id, partition)
208+
for envelope, (stage_id, partition) in zip(envelopes, tasks, strict=True)
176209
]
177210

178-
worker_rows: dict[int, int] = {}
211+
task_rows: dict[tuple[int, int], int] = {}
179212
failures = []
180-
for partition, worker in zip(partitions, workers, strict=True):
213+
for task, worker in zip(tasks, workers, strict=True):
181214
stdout, stderr = worker.communicate()
182215
if worker.returncode != 0:
183-
failures.append(f"partition {partition} failed:\n{stderr}")
216+
stage_id, partition = task
217+
failures.append(f"stage {stage_id} partition {partition} failed:\n{stderr}")
184218
continue
185-
worker_rows[partition] = json.loads(stdout)["rows"]
219+
task_rows[task] = json.loads(stdout)["rows"]
186220

187221
if failures:
188222
raise RuntimeError("\n".join(failures))
189223

190224
# Now run the whole query here. Every stage partition has a file, so the
191-
# stage node streams them instead of recomputing -- the driver does only
225+
# stage nodes stream them instead of recomputing -- the driver does only
192226
# the final merge.
193227
batches = ctx.sql(sql).collect()
194-
_ = engine
195-
return DistributedResult(batches, partitions, worker_rows)
228+
return DistributedResult(batches, tasks, task_rows)
196229

197230

198231
def run_local(

‎examples/distributed/engine-library/python/tests/_test_three_libraries.py‎

Lines changed: 66 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@
3434
import pytest
3535
from datafusion import SessionContext, udf
3636
from dfx_engine import _internal
37-
from dfx_engine.driver import find_stage, run_distributed, run_local
37+
from dfx_engine.driver import find_stage, find_stages, run_distributed, run_local
3838
from dfx_engine.session import SessionSpec, build_session, expected_codec_ids
3939
from dfx_engine.worker import run_task
4040

@@ -83,7 +83,8 @@ def test_distributed_aggregate_matches_single_process(spec: SessionSpec) -> None
8383
"""Query 1: the baseline. Four input files, four workers, one answer."""
8484
result = run_distributed(Q1, spec)
8585

86-
assert result.partitions == [0, 1, 2, 3]
86+
# One stage, so every task is a partition of stage 1.
87+
assert result.tasks == [(1, 0), (1, 1), (1, 2), (1, 3)]
8788
assert _rows(result.batches) == _rows(run_local(Q1, spec))
8889
# Checked by hand against the fixture.
8990
assert _rows(result.batches) == [
@@ -211,11 +212,11 @@ def test_every_partition_ran_exactly_once(spec: SessionSpec) -> None:
211212
"""Each worker got a different partition, and together they covered it."""
212213
result = run_distributed(Q1, spec)
213214

214-
assert sorted(result.partitions) == [0, 1, 2, 3]
215-
assert len(set(result.partitions)) == len(result.partitions)
215+
assert sorted(result.tasks) == [(1, 0), (1, 1), (1, 2), (1, 3)]
216+
assert len(set(result.tasks)) == len(result.tasks)
216217
# Two rows per input file, so each worker saw two rows' worth of groups.
217-
assert sum(result.worker_rows.values()) == 8
218-
assert set(result.worker_rows) == set(result.partitions)
218+
assert sum(result.task_rows.values()) == 8
219+
assert set(result.task_rows) == set(result.tasks)
219220

220221

221222
def test_each_worker_published_its_own_file(spec: SessionSpec) -> None:
@@ -231,6 +232,8 @@ def test_each_worker_published_its_own_file(spec: SessionSpec) -> None:
231232
for partition in range(4)
232233
)
233234
assert produced == expected
235+
# Nothing half-written: a temporary file is unique to its writer and is
236+
# renamed away when the partition is complete.
234237
assert list(shuffle.glob("*.tmp")) == []
235238

236239

@@ -295,6 +298,63 @@ def test_the_plan_splits_at_the_partial_aggregate(spec: SessionSpec) -> None:
295298
assert stage.output_partitioning.scheme == "UnknownPartitioning"
296299

297300

301+
TWO_BRANCHES = """
302+
select l_returnflag as g, sum(l_quantity) as qty
303+
from lineitem group by l_returnflag
304+
union all
305+
select l_linestatus as g, sum(l_quantity) as qty
306+
from other group by l_linestatus
307+
"""
308+
309+
310+
def test_two_branches_get_two_separately_numbered_stages(
311+
lineitem_dir: pathlib.Path, tmp_path: pathlib.Path
312+
) -> None:
313+
"""A partial aggregate in each branch is two stages, not one.
314+
315+
Both branches are independent subtrees and both want shipping, so the
316+
planner wraps each. They must not share a stage id: a stage exchanges
317+
results through paths built from that id, so two stages numbered alike
318+
would write to the same files -- and because a union drives its branches
319+
concurrently, they would do it at the same time. That fails outright
320+
rather than quietly, but only because the schemas happen to differ.
321+
"""
322+
spec = SessionSpec(
323+
tables={"lineitem": str(lineitem_dir), "other": str(lineitem_dir)},
324+
shuffle_dir=str(tmp_path / "shuffle"),
325+
target_partitions=2,
326+
)
327+
328+
ctx, engine, _storage = build_session(spec)
329+
plan = ctx.sql(TWO_BRANCHES).execution_plan()
330+
331+
assert plan.display_indent().count("ShuffleStageExec") == 2
332+
assert len(find_stages(plan)) == 2
333+
assert engine.stages_inserted() == 2
334+
335+
result = run_distributed(TWO_BRANCHES, spec)
336+
337+
# Four partitions apiece, under two distinct stage ids.
338+
assert sorted(result.tasks) == [
339+
(stage, part) for stage in (1, 2) for part in range(4)
340+
]
341+
# Sorted: a union has no ordering of its own, so the branches interleave
342+
# differently from run to run.
343+
assert sorted(_rows(result.batches)) == sorted(_rows(run_local(TWO_BRANCHES, spec)))
344+
345+
# Each stage published its own files, so neither read the other's.
346+
produced = sorted(
347+
path.name for path in pathlib.Path(spec.shuffle_dir).glob("*.arrow")
348+
)
349+
assert produced == sorted(
350+
pathlib.Path(
351+
_internal.partition_path(spec.shuffle_dir, _internal.stage_id(index), part)
352+
).name
353+
for index in range(2)
354+
for part in range(4)
355+
)
356+
357+
298358
# --- the ways it goes wrong -------------------------------------------------
299359

300360

‎examples/distributed/engine-library/src/lib.rs‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@
2828
//!
2929
//! One of three libraries in `examples/distributed`. This one owns execution.
3030
31+
use pyo3::exceptions::PyValueError;
3132
use pyo3::prelude::*;
3233

3334
use crate::config::DfxEngineConfig;
@@ -52,10 +53,19 @@ fn partition_path(shuffle_dir: &str, stage_id: u32, partition: usize) -> String
5253
.into_owned()
5354
}
5455

55-
/// The stage id this engine's planner produces.
56+
/// Id of the `index`th stage in a plan, counting in pre-order from the root.
57+
///
58+
/// Exported for the same reason as [`partition_path`]: a foreign node's
59+
/// one-line display is replaced by the FFI wrapper's, so Python cannot read a
60+
/// stage id back off a plan and has to agree with Rust about the numbering
61+
/// instead. Keeping the arithmetic here means the two cannot drift.
5662
#[pyfunction]
57-
fn stage_id() -> u32 {
58-
planner::STAGE_ID
63+
#[pyo3(signature = (index=0))]
64+
fn stage_id(index: usize) -> PyResult<u32> {
65+
u32::try_from(index)
66+
.ok()
67+
.and_then(|index| planner::FIRST_STAGE_ID.checked_add(index))
68+
.ok_or_else(|| PyValueError::new_err(format!("stage index {index} is out of range")))
5969
}
6070

6171
#[pymodule]

0 commit comments

Comments
 (0)