replay_does_not_feed_http_input_during_commit: checkpoint fails with NoPersistentId on balanced AccumulateZ1Trace, circuit terminates #7631

Verdict: pre-existing bug on main — not caused by PR #7628 (which only adds a SQL-compiler regression test, see git diff origin/main...issue7212 --stat).

controller::test::replay_does_not_feed_http_input_during_commit fails during the merge-queue run for #7628, and had already failed the same way on a prior merge-queue attempt of the same PR, on the other CPU architecture — so it isn't an arch-specific flake either.

Evidence

In both logs, right after replay finishes, checkpointing starts failing every 60s:

WARN dbsp_adapters::controller: checkpoint failed: DBSP error: storage error: Internal error: operator AccumulateZ1Trace nn16 has not been assigned a persistent id.
WARN dbsp_adapters::controller: checkpoint failed: DBSP error: runtime error: circuit has been terminated

(arm64 log lines 2510, 2513+). RSS then balloons from ~4 GiB to ~14 GiB over the following minutes before the circuit dies and the test's final assert_eq!(count_rows(...), RECORDS + 1) fails because the last record never got processed.

AccumulateZ1Trace is created by the join/aggregate balancer (crates/dbsp/src/operator/dynamic/balance/accumulate_trace_balanced.rs) when weighted_count()'s skewed-key aggregate in test_circuit_with_aggregate (crates/adapters/src/test.rs) triggers rebalancing — the test's skewed key distribution is deliberately designed to do that. The dynamically-created accumulate-trace operator (nn16) apparently isn't getting a persistent id assigned, so require_persistent_id fails on every checkpoint attempt once balancing kicks in, which looks like it then runs away (repeated failures + large RSS growth) until the circuit is terminated.

The test and its host circuit (replay_does_not_feed_http_input_during_commit, test_circuit_with_aggregate) were added/last touched Oct 20 by commit 9c4b63e, unrelated to #7628.

Next step

Look at persistent-id propagation for the balancer's dynamically-created AccumulateZ1Trace/AccumulateTraceAppend operators in accumulate_trace_balanced.rs — it seems the balanced accumulator doesn't inherit/derive a persistent id from its upstream aggregate the way checkpointing expects, only surfacing once skew triggers rebalancing mid-run.

Requeuing #7628 is likely to hit the same failure again since it's unrelated to the PR's content.