BUG: re-raise KeyboardInterrupt after an interrupted Monte Carlo run - #1177
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #1177 +/- ##
===========================================
+ Coverage 91.50% 91.51% +0.01%
===========================================
Files 131 131
Lines 17723 17733 +10
===========================================
+ Hits 16217 16229 +12
+ Misses 1506 1504 -2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
thc1006
left a comment
There was a problem hiding this comment.
Why include these three empty files ?:
monte_carlo_test.errors.txtmonte_carlo_test.inputs.txtmonte_carlo_test.outputs.txt
Could you drop them from the PR?
Can use AI but plz check the claim against the implementation by yourself
| # pylint: disable=broad-except | ||
| except (Exception, KeyboardInterrupt) as error: | ||
| except (Exception, KeyboardInterrupt): | ||
| simulation_error_event.set() |
There was a problem hiding this comment.
The cleanup try starts only after every worker has been created and started. A Ctrl-C during Process.start() skips the event/set and join path, so any workers already started are not explicitly stopped or joined.
Could the startup loop be covered by the same cleanup path, with a separate list of processes whose start() has completed?
There was a problem hiding this comment.
Done, with the started-list shape you suggested. The startup loop is inside the cleanup try, and each worker joins the list only after its start() returns, so the handler never joins a process that was never started (which raises).
test_ctrl_c_during_worker_startup_still_stops_the_started_workers pins it: the second worker's start() raises where a real Ctrl-C could land, and the test asserts the first worker was signalled and joined while the never-started one was not (joins == 0). Reverting only this restructuring makes that test fail; the other seven still pass.
The narrow edge that remains is an interrupt inside start() after the OS process exists but before start() returns — that worker is not in the list and is not cleaned up. I did not find a way to close that without reaching into multiprocess internals, so it is left as stated.
| def test_interrupted_serial_run_reaches_the_caller(tmp_path): | ||
| """``simulate`` used to return normally after Ctrl-C. | ||
|
|
||
| A caller could not tell a partial run from a complete one without opening | ||
| the output file and counting rows. | ||
| """ | ||
| mc = _InterruptingMonteCarlo(str(tmp_path / "run"), interrupt_after=2) | ||
|
|
||
| with pytest.raises(KeyboardInterrupt): | ||
| mc.simulate(number_of_simulations=10, parallel=False) | ||
|
|
||
|
|
||
| def test_interrupted_serial_run_keeps_the_rows_that_finished(tmp_path): | ||
| """The two simulations that completed stay readable and paired.""" | ||
| mc = _InterruptingMonteCarlo(str(tmp_path / "run"), interrupt_after=2) | ||
|
|
||
| with pytest.raises(KeyboardInterrupt): | ||
| mc.simulate(number_of_simulations=10, parallel=False) | ||
|
|
||
| inputs = (tmp_path / "run.inputs.txt").read_text(encoding="utf-8").splitlines() | ||
| outputs = (tmp_path / "run.outputs.txt").read_text(encoding="utf-8").splitlines() | ||
|
|
||
| assert len(inputs) == 2 | ||
| assert len(outputs) == 2 | ||
| assert [json.loads(row)["index"] for row in inputs] == [ | ||
| json.loads(row)["index"] for row in outputs | ||
| ] | ||
|
|
||
|
|
||
| def test_interrupted_serial_run_still_reloads_the_logs(tmp_path): |
There was a problem hiding this comment.
test_interrupted_serial_run_still_reloads_the_logs overrides __terminate_simulation and only checks a boolean. That proves the method was called, but not that the real setters reload the rows or update num_of_loaded_sims.
Could this exercise the real termination path and assert the loaded input/output rows and the next append=True call? The distinction matters because parallel interruptions can leave gaps, which is the row-count problem tracked in #1075.
There was a problem hiding this comment.
Done. The stub is gone from every serial test: they run the real __terminate_simulation and assert the state the reload produces — num_of_loaded_sims == 2, two rows in each log, results populated — and test_an_interrupted_run_can_be_continued_with_append runs the documented path end to end: interrupt at 2 of 10, simulate(..., append=True), then asserts the indices on disk are 1–10 with no gap and no repeat.
On the #1075 connection: agreed that parallel interruptions leave gaps, which is why the append continuation is only asserted for serial mode here. The parallel tests assert the reload ran and the interrupt propagated, not that a parallel resume is gap-free — that stays #1075's problem.
|
After the first The fake worker always finishes on its second join, so the regression test cannot detect this case. Should this use the bounded fleet-shutdown path being split from #1054, or should the PR narrow its guarantee to workers that exit cooperatively? |
|
There is still an interrupt-sized hole between this PR and #1125: Ctrl-C after the input append but before the output append completes can therefore leave a one-sided record. The following |
b5cf7eb to
bd50673
Compare
|
Both points are fair. Dropped, and I went back over the claims. The three filesGone — the branch is now They were never meant to be here. Two force-pushes rather than one: my first amend dropped them, then I re-ran the suite before pushing, which recreated them, and Checking the claims against the implementationYou were right to push on this. The claim in the description is that Replaced. The stub is gone, the tests run the real method, and they assert the state it produces: assert mc.num_of_loaded_sims == 2
assert len(mc.inputs_log) == 2
assert len(mc.outputs_log) == 2
assert mc.results["apogee"] == [1001.0, 1002.0]And a sixth test that runs the documented path end to end rather than asserting about it — interrupt after two of ten, then continue: mc.interrupt_after = 10
mc.simulate(number_of_simulations=10, append=True, parallel=False)
rows = pathlib.Path(stem + ".outputs.txt").read_text(encoding="utf-8").splitlines()
assert [json.loads(row)["index"] for row in rows] == list(range(1, 11))That passes: ten rows, indices 1 to 10, no gap and no repeat. That the tests pin anythingReverting only With the change: The scope note in the description still stands: |
thc1006
left a comment
There was a problem hiding this comment.
I still cannot approve this head yet. The startup window is unchanged: worker creation and start() still happen before the cleanup try, so an interrupt while starting a later worker leaves the already-started workers outside the signal/join path. The second round of joins is also still unbounded, so a worker stuck in a simulation can prevent the original interrupt from ever reaching the caller.
There is one more direct path through _append_simulation_record: it catches Exception, not BaseException, and only rolls back the input file. A KeyboardInterrupt during an output write can therefore leave a one-sided or partial record; the reload in simulate() can then replace the interrupt with JSONDecodeError. I think that boundary needs a regression test and a two-file rollback before this closes #1151.
Also, after a record is successfully appended, an interrupt from print_update_status() writes that already-committed input row to the error file because inputs_json has not been cleared yet.
The current Actions run is green across all six OS/Python legs, Codecov, lint and docs. The remaining concerns are lifecycle and write-boundary cases that the fake-worker test does not exercise.
If you're using AI to generate PRs, could you please open them as Drafts first? Give them a thorough self-review, and maybe even prompt an AI to do an adversarial review on your PR. Iterate on this a few times until everything looks solid, and only then mark it as 'Ready for review'. Thanks!
bd50673 to
8ca651f
Compare
|
All four points are addressed at the new head; the description was rewritten to match. Taking the two thread-level ones here. The unbounded second joinYou are right that the regression test could not detect a stuck worker —
If #1054 lands first, the re-raise here rebases onto its bounded path and inherits the stronger guarantee; if this lands first, #1054 replaces the unbounded join it finds. The
|
8ca651f to
75ac771
Compare
|
Your comment and my previous push crossed within the same minute — the head you reviewed was The committed row written to the error fileConfirmed at my head too, and your diagnosis was exact: after a successful The two-file rollbackTaken as you described rather than as regression-test-only, because the interrupt-point audit showed the one-file version has a second failure mode: an interrupt inside either write (not just between them) leaves a torn partial row, and the reload then dies with
Startup window and the second join, at this headBoth were in Verification at
|
|
One correction to my previous comment before anyone relies on it. I wrote "each of the five fixes was also reverted individually with only its own tests failing." When I posted that, I had individually reverted four of the five, each verified with a
So the substantive point stands — every fix is pinned by at least one test that fails without it, all ten fail at |
simulate() returned normally after Ctrl-C. Both execution modes caught the interrupt and neither re-raised it, so simulate() went on to __terminate_simulation() and returned exactly as it does after a complete study. A caller could not tell a partial run from a finished one without opening the output file and counting rows. Five more holes sat next to that one, found in review and in the interrupt-point audit it prompted: - __run_in_serial bound inputs_json inside the loop body, so Ctrl-C during the first keep_simulating() call reached the handler with nothing bound and the run died with UnboundLocalError from inside the cleanup. - It also never cleared inputs_json after a successful append, so Ctrl-C landing in the progress print — or in the next keep_simulating() call — wrote the row that had just committed into the error file as though it never finished. - _append_simulation_record rolled back on Exception, and KeyboardInterrupt does not derive from Exception, so Ctrl-C between the two appends left a one-sided record. - Its rollback also only truncated the inputs file, so an interrupt inside either write left a torn partial row for the reload to die on with JSONDecodeError instead of the interrupt. - The parallel cleanup began only after every worker had started, so Ctrl-C during the startup loop left the already-started workers running with nobody signalling or joining them. Bind inputs_json before the loop and clear it after each committed append, roll both files back on BaseException best-effort, cover the startup loop with the same cleanup path over a started-workers list, and re-raise in both handlers. Catch the interrupt in simulate() so __terminate_simulation() still runs before it leaves: it reloads the logs through the file setters, and set_num_of_loaded_sims is what the documented append=True continuation reads. The shutdown join stays unbounded, and the docstring now says so: the interrupt propagates once every worker finishes the simulation it is in and exits on its own. A worker stuck inside one simulation blocks the interrupt as it already blocked the run; killing it here could tear a half-written row into logs it holds the mutex for, and the bounded fleet shutdown belongs to RocketPy-Team#1054. The ordinary exception path is unchanged. Add ten regression tests covering both modes, the early interrupt, the preserved rows, the reload, an interrupted run continued with append=True, the between-appends rollback, the torn-write rollback, the committed row staying out of the error file, and the startup-loop cleanup. They run the real __terminate_simulation and assert the state it produces; all fail on develop, and each fix was also reverted individually with only its own test failing.
75ac771 to
138825b
Compare
__sim_producer binds sim_idx and inputs_json inside the loop, and its handler reads both. A worker that fails in the seeding above the loop, or in the claim that opens it, reached the handler with neither name assigned and died with UnboundLocalError instead of recording anything. The parent learns a worker failed from error_event, which the handler sets on its last line, so it was never reached either and the run waited rather than stopping. Both names are bound before the try now, and the message says worker startup when no index was claimed rather than naming one that does not exist. Binding only the name in the traceback is not enough: the message then raises on the other one, which the tests cover. Scoped to the parallel producer. The serial handler has the same unbound inputs_json and is RocketPy-Team#1177's. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
__sim_producer binds sim_idx and inputs_json inside the loop, and its handler reads both. A worker that fails in the seeding above the loop, or in the claim that opens it, reached the handler with neither name assigned and died with UnboundLocalError instead of recording anything. The parent learns a worker failed from error_event, which the handler sets on its last line, so it was never reached either and the run waited rather than stopping. Both names are bound before the try now, and the message says worker startup when no index was claimed rather than naming one that does not exist. Binding only the name in the traceback is not enough: the message then raises on the other one, which the tests cover. Scoped to the parallel producer. The serial handler has the same unbound inputs_json and is RocketPy-Team#1177's. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
__sim_producer binds sim_idx and inputs_json inside the loop, and its handler reads both. A worker that fails in the seeding above the loop, or in the claim that opens it, reached the handler with neither name assigned and died with UnboundLocalError instead of recording anything. The parent learns a worker failed from error_event, which the handler sets on its last line, so it was never reached either and the run waited rather than stopping. Both names are bound before the try now, and the message says worker startup when no index was claimed rather than naming one that does not exist. Binding only the name in the traceback is not enough: the message then raises on the other one, which the tests cover. Scoped to the parallel producer. The serial handler has the same unbound inputs_json and is RocketPy-Team#1177's. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
__sim_producer binds sim_idx and inputs_json inside the loop, and its handler reads both. A worker that fails in the seeding above the loop, or in the claim that opens it, reached the handler with neither name assigned and died with UnboundLocalError instead of recording anything. The parent learns a worker failed from error_event, which the handler sets on its last line, so it was never reached either and the run waited rather than stopping. Both names are bound before the try now, and the message says worker startup when no index was claimed rather than naming one that does not exist. Binding only the name in the traceback is not enough: the message then raises on the other one, which the tests cover. Scoped to the parallel producer. The serial handler has the same unbound inputs_json and is RocketPy-Team#1177's. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
|
Reading this while working next door in #1182, and two things seemed worth passing on. It is your draft, so treat both as information rather than requests. The So with We have each fixed the same bug on opposite paths, and disagree in one place. You reset The one real conflict is #1182 is approved and unmerged at the moment, so there is still time either way. |
Six ways a worker that failed got away without saying so. It died inside its own handler on an unbound name, so the manager lock it was holding was never given back and the run waited for good. A worker that was killed ran no handler, set no event, and left only an exit code nobody read, so simulate() returned normally with rows missing. Reporting a failure could itself block on a lock a dead sibling still held. Both names are bound before the try now, the handler reports through a bounded lock and releases it from a finally, and the parent reads exit codes and the failure event instead of joining unbounded. A run that is only slow is still never cut short: how a worker ended decides that, not how long it took. An exit code cannot show a worker that left between claiming an index and recording it, so a run is checked against its own logs at the end. Both must hold exactly the simulations asked for, none twice, none unreadable. Scope is the parallel producer. __run_in_serial has the same unbound name and belongs to RocketPy-Team#1177. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Six ways a worker that failed got away without saying so. It died inside its own handler on an unbound name, so the manager lock it was holding was never given back and the run waited for good. A worker that was killed ran no handler, set no event, and left only an exit code nobody read, so simulate() returned normally with rows missing. Reporting a failure could itself block on a lock a dead sibling still held. Both names are bound before the try now, the handler reports through a bounded lock and releases it from a finally, and the parent reads exit codes and the failure event instead of joining unbounded. A run that is only slow is still never cut short: how a worker ended decides that, not how long it took. An exit code cannot show a worker that left between claiming an index and recording it, so a run is checked against its own logs at the end. Both must hold exactly the simulations asked for, none twice, none unreadable. Scope is the parallel producer. __run_in_serial has the same unbound name and belongs to #1177. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
# Conflicts: # rocketpy/simulation/monte_carlo.py
0923d8a
into
RocketPy-Team:develop
…ial loop The rebase onto develop was clean in three files and left one add/add conflict in __run_in_serial: both sides bind a name before the try so the handlers cannot meet it unbound. RocketPy-Team#1177 binds inputs_json for the KeyboardInterrupt handler, this branch binds sim_idx for the Exception one. Both are kept; dropping either puts back the UnboundLocalError the other side had just removed. Keeping both then pushed the function to 27 statements, two over max-statements, so the error-file append the two handlers had copies of moved into __record_failed_inputs. The guard travels with it: the error file is created in __setup_files, so appending "" to it was already a no-op, and the unconditional copy and the guarded one did the same thing. What the merge could not show is that RocketPy-Team#1177's tests were written against the loop this branch replaces: - _InterruptingMonteCarlo stubs only what __run_in_serial touched, and the loop now also calls __seed_this_simulation, which reads models the double has none of. Stubbed, like the other name-mangled members; seeding is pinned by test_monte_carlo_seeding.py. - Its rows carried no run_root, so the append test tripped the guard for studies written before that check existed instead of exercising the continuation. The double now writes the root the way the real row builders do, the inputs file whole and the outputs file by digest. - _SimMonitor.keep_simulating is now claim_next_index, and one test monkeypatches it by name. - Serial numbering now starts at zero, as the parallel path always did, so the three tests reading indices off disk expected 1, 2, 3 where a run writes 0, 1, 2. tests/unit/simulation: 322 passed, 8 skipped. ruff clean, pylint 10.00. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ial loop The rebase onto develop was clean in three files and left one add/add conflict in __run_in_serial: both sides bind a name before the try so the handlers cannot meet it unbound. RocketPy-Team#1177 binds inputs_json for the KeyboardInterrupt handler, this branch binds sim_idx for the Exception one. Both are kept; dropping either puts back the UnboundLocalError the other side had just removed. Keeping both then pushed the function to 27 statements, two over max-statements, so the error-file append the two handlers had copies of moved into __record_failed_inputs. The guard travels with it: the error file is created in __setup_files, so appending "" to it was already a no-op, and the unconditional copy and the guarded one did the same thing. What the merge could not show is that RocketPy-Team#1177's tests were written against the loop this branch replaces: - _InterruptingMonteCarlo stubs only what __run_in_serial touched, and the loop now also calls __seed_this_simulation, which reads models the double has none of. Stubbed, like the other name-mangled members; seeding is pinned by test_monte_carlo_seeding.py. - Its rows carried no run_root, so the append test tripped the guard for studies written before that check existed instead of exercising the continuation. The double now writes the root the way the real row builders do, the inputs file whole and the outputs file by digest. - _SimMonitor.keep_simulating is now claim_next_index, and one test monkeypatches it by name. - Serial numbering now starts at zero, as the parallel path always did, so the three tests reading indices off disk expected 1, 2, 3 where a run writes 0, 1, 2. tests/unit/simulation: 322 passed, 8 skipped. ruff clean, pylint 10.00. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ial loop The rebase onto develop was clean in three files and left one add/add conflict in __run_in_serial: both sides bind a name before the try so the handlers cannot meet it unbound. RocketPy-Team#1177 binds inputs_json for the KeyboardInterrupt handler, this branch binds sim_idx for the Exception one. Both are kept; dropping either puts back the UnboundLocalError the other side had just removed. Keeping both then pushed the function to 27 statements, two over max-statements, so the error-file append the two handlers had copies of moved into __record_failed_inputs. The guard travels with it: the error file is created in __setup_files, so appending "" to it was already a no-op, and the unconditional copy and the guarded one did the same thing. What the merge could not show is that RocketPy-Team#1177's tests were written against the loop this branch replaces: - _InterruptingMonteCarlo stubs only what __run_in_serial touched, and the loop now also calls __seed_this_simulation, which reads models the double has none of. Stubbed, like the other name-mangled members; seeding is pinned by test_monte_carlo_seeding.py. - Its rows carried no run_root, so the append test tripped the guard for studies written before that check existed instead of exercising the continuation. The double now writes the root the way the real row builders do, the inputs file whole and the outputs file by digest. - _SimMonitor.keep_simulating is now claim_next_index, and one test monkeypatches it by name. - Serial numbering now starts at zero, as the parallel path always did, so the three tests reading indices off disk expected 1, 2, 3 where a run writes 0, 1, 2. tests/unit/simulation: 322 passed, 8 skipped. ruff clean, pylint 10.00. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ial loop The rebase onto develop was clean in three files and left one add/add conflict in __run_in_serial: both sides bind a name before the try so the handlers cannot meet it unbound. RocketPy-Team#1177 binds inputs_json for the KeyboardInterrupt handler, this branch binds sim_idx for the Exception one. Both are kept; dropping either puts back the UnboundLocalError the other side had just removed. Keeping both then pushed the function to 27 statements, two over max-statements, so the error-file append the two handlers had copies of moved into __record_failed_inputs. The guard travels with it: the error file is created in __setup_files, so appending "" to it was already a no-op, and the unconditional copy and the guarded one did the same thing. What the merge could not show is that RocketPy-Team#1177's tests were written against the loop this branch replaces: - _InterruptingMonteCarlo stubs only what __run_in_serial touched, and the loop now also calls __seed_this_simulation, which reads models the double has none of. Stubbed, like the other name-mangled members; seeding is pinned by test_monte_carlo_seeding.py. - Its rows carried no run_root, so the append test tripped the guard for studies written before that check existed instead of exercising the continuation. The double now writes the root the way the real row builders do, the inputs file whole and the outputs file by digest. - _SimMonitor.keep_simulating is now claim_next_index, and one test monkeypatches it by name. - Serial numbering now starts at zero, as the parallel path always did, so the three tests reading indices off disk expected 1, 2, 3 where a run writes 0, 1, 2. tests/unit/simulation: 322 passed, 8 skipped. ruff clean, pylint 10.00. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ith (#1187) * ENH: continue a Monte Carlo study from the root its rows were drawn with A Monte Carlo run cannot be seeded on develop: simulate() takes no random_seed. Seeding it per simulation index is not enough on its own, because an append then derives a fresh root and writes it into the same file, so a study resumed after a restart holds two lineages with nothing afterwards to say which simulation came from which. Both halves are here. A simulation takes its seed from its own index, so a serial run and a run split over workers draw the same inputs for the same index. Every input row records the root that drew it, and an append reads it back rather than needing to be given it again. A seed that disagrees with the rows is refused, as is a log whose rows disagree with each other, and one whose rows carry no root at all, which is how a log written before this looks. Output rows carry a digest of that root, so a log belonging to another study is refused even when its indices line up with this one's. The worker tests in #1182 drive the producer with a stand-in monitor, so they move to the claim along with it. A reseed failure now names the index it was seeding for rather than worker startup, because the seeding happens after the claim rather than once above the loop. The seeding half was #1054, closed in favour of this. Addresses #1053 and #1075. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * TST: carry #1177's interrupt tests onto the reseeded serial loop The rebase onto develop was clean in three files and left one add/add conflict in __run_in_serial: both sides bind a name before the try so the handlers cannot meet it unbound. #1177 binds inputs_json for the KeyboardInterrupt handler, this branch binds sim_idx for the Exception one. Both are kept; dropping either puts back the UnboundLocalError the other side had just removed. Keeping both then pushed the function to 27 statements, two over max-statements, so the error-file append the two handlers had copies of moved into __record_failed_inputs. The guard travels with it: the error file is created in __setup_files, so appending "" to it was already a no-op, and the unconditional copy and the guarded one did the same thing. What the merge could not show is that #1177's tests were written against the loop this branch replaces: - _InterruptingMonteCarlo stubs only what __run_in_serial touched, and the loop now also calls __seed_this_simulation, which reads models the double has none of. Stubbed, like the other name-mangled members; seeding is pinned by test_monte_carlo_seeding.py. - Its rows carried no run_root, so the append test tripped the guard for studies written before that check existed instead of exercising the continuation. The double now writes the root the way the real row builders do, the inputs file whole and the outputs file by digest. - _SimMonitor.keep_simulating is now claim_next_index, and one test monkeypatches it by name. - Serial numbering now starts at zero, as the parallel path always did, so the three tests reading indices off disk expected 1, 2, 3 where a run writes 0, 1, 2. tests/unit/simulation: 322 passed, 8 skipped. ruff clean, pylint 10.00. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * DOC: record the serial renumbering and the refused append as changes The entry this branch added sits under Added, and both of these are visible to someone whose code already reads a study off disk: a serial run now numbers its simulations from zero, so the index field shifts by one, and an append onto a study written before this release is refused rather than continued. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * DOC: say how a run numbers its simulations Reviewing the Changed entry @Gui-FernandesBR added turned up a gap on my side: the serial renumbering is a break for anyone reading indices off disk, and the only place it was written down was the changelog. The guide's note leans on it without saying so. It claims simulation 7 draws the same inputs serially or split over workers, which is only one claim if both paths agree on which simulation 7 is, so the note now says a run of n numbers them 0 to n - 1. The notebook says it where a reader meets simulate(). Its stored outputs predate this and are left alone: nbsphinx never re-executes them, and a seeding change moves every number in them, which is not a diff worth reading. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * TST: take the wall clock out of the shared-deadline test This branch's CI went red on windows-latest 3.10 with assert min(offered) < 0.05 E assert 0.0500000000001819 < 0.05 The test lives on develop rather than in this branch, and #1182 is where I added it. The stand-in workers return at once, so nothing makes the clock move between _wait_for_the_workers setting its deadline and reading it back for the last of them, and (t + 0.05) - t is not exactly 0.05 in binary. Every other leg passed, and so did the same job on 3.14. The clock is a counter now, so what each worker is offered is decided by arithmetic rather than by how coarse the platform's timer is. The shape the test is about is asserted as well: inside one stage the offers shrink along the fleet, where one deadline each would hand every worker the whole grace. Giving each worker its own grace turns both assertions red and leaves the other twelve in the file green. It sits here because it blocks this pull request's CI, and it is a test-only change to a file this branch does not otherwise touch. Happy to move it to its own pull request if you would rather keep the two apart. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> --------- Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> Co-authored-by: Gui-FernandesBR <guilherme_fernandes@usp.br> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
BUG: re-raise KeyboardInterrupt after a Monte Carlo run is interrupted
Closes #1151.
The problem
MonteCarlo.simulate()returned normally after Ctrl-C. Both execution modes caught the interrupt and neither re-raised it, sosimulate()went on to__terminate_simulation()and returned exactly as it does after a complete study. A caller could not tell a partial run from a finished one without opening the output file and counting rows, and a command-line caller could not return the conventional interrupted status.At
4263fa95d7fe6f63d9593f01e4ff7a088369e195, asking for ten simulations and interrupting the third:Review on this pull request, and the interrupt-point audit it prompted, found five more holes sitting next to that one; this branch now fixes all six.
inputs_jsonunbound on an early interrupt.__run_in_serialbinds it inside the loop body, so Ctrl-C during the firstkeep_simulating()call reached the handler with nothing bound:A one-sided record on an interrupt between the two appends.
_append_simulation_recordrolled its inputs row back onException, andKeyboardInterruptderives fromBaseException, so Ctrl-C after the inputs append but before the outputs append left the inputs file one row longer than the outputs file — which is the pairing BUG: write MonteCarlo input and output rows atomically (#1110) #1125 exists to protect. Reproduced before the fix: 2 inputs rows against 1 outputs row.A committed row written to the error file. After a successful append,
inputs_jsonstill held the committed row, so an interrupt landing in the progress print — or in the nextkeep_simulating()orincrement()call, or inprint_final_status()after the last row — wrote a row that had already committed into the error file as though it never finished.A torn partial row surviving the rollback. The rollback only truncated the inputs file, so an interrupt inside either write (not just between them) left half a row on disk, and the reload then died with
JSONDecodeErrorin place of the interrupt.No cleanup for workers started before an interrupt in the startup loop. The parallel cleanup
trybegan only after every worker had been created and started, so Ctrl-C duringProcess.start()left the already-started workers running with nobody signalling or joining them.The change
All in
rocketpy/simulation/monte_carlo.py, no behavior removed:__run_in_serialbindsinputs_jsonbefore the loop, and re-raises after writing the unfinished record._append_simulation_recordputs both writes in onetryand, onBaseException, truncates both files back to their entry sizes — best-effort, so the rollback cannot replace the original failure with its own. The interrupt gets the same rollback as any other failure, the two files stay paired, and no torn row survives for the reload to trip over.__run_in_serialclearsinputs_jsonas soon as the append returns, so an interrupt after the commit reports nothing rather than reporting the committed row as unfinished.__run_in_parallelstarts the workers inside the cleanuptry, appending each to the process list only after itsstart()returns, so an interrupt mid-startup still signals and joins whatever came up — and never joins a process that was never started, which raises. The handler re-raises unconditionally; only the re-raise was conditional before, so a bareraisecovers both failure kinds and drops theraise errorrebinding.simulate()catches the interrupt, runs__terminate_simulation(), then re-raises. That call reloads the logs through the file property setters, andset_num_of_loaded_simsis what the documentedappend=Truecontinuation reads; re-raising past it would leave the object disagreeing with its own files. The ordinary exception path is untouched.The docstring gains a
Raisessection that states the guarantee exactly, including its limit.What this guarantees, and what it deliberately does not
In parallel mode the interrupt propagates after every worker finishes the simulation it is currently in, notices the stop event, and exits on its own. The shutdown join stays unbounded, so a worker stuck inside one simulation blocks the interrupt exactly as it already blocked the run on
develop.That is a narrower guarantee than "Ctrl-C always returns promptly", and it is intentional. Workers write their rows under the shared mutex; killing one from the outside after a timeout can tear a half-written row into the logs and leave the mutex held, which trades a hang for corrupted files. The bounded fleet shutdown is #1054's design territory, and this pull request should not grow a competing one. The docstring and the fake-worker docstring in the tests both state the limit rather than implying coverage the tests do not have.
Verification
Same two reproductions at the head of this branch:
Ten regression tests in
tests/unit/simulation/test_monte_carlo.py:test_interrupted_serial_run_reaches_the_callerKeyboardInterrupttest_interrupted_serial_run_keeps_the_rows_that_finishedtest_interrupted_serial_run_still_reloads_the_logsnum_of_loaded_sims == 2, both logs hold 2 rows,resultspopulatedtest_an_interrupted_run_can_be_continued_with_appendappend=True, indices on disk are 1–10 with no gap and no repeattest_ctrl_c_before_the_first_simulation_is_still_the_interruptUnboundLocalErrortest_ctrl_c_between_the_two_appends_rolls_the_inputs_row_backtest_interrupted_parallel_run_signals_joins_and_reaches_the_callertest_ctrl_c_during_worker_startup_still_stops_the_started_workersProcess.start()still signals and joins the started worker, and does not join the never-started onetest_ctrl_c_in_the_progress_print_leaves_the_error_file_emptytest_a_torn_outputs_write_rolls_both_files_backThe serial tests run the real
__terminate_simulationrather than a stub and assert the state it produces. The parallel tests drive__run_in_parallelover stubs for_import_multiprocessand_create_multiprocess_manager, so they start no processes and stay deterministic in the default suite; per the section above, they cover workers that exit cooperatively, and say so.All ten fail at
develop— the whole file run, no filter:10 failed, 65 passed, with every pre-existing test still green. Each of the five change bullets was then reverted individually and the whole file re-run:BaseExceptionrollbacktry+ unconditional parallel re-raisesimulate()catch + reload + re-raiseappend=Truecontinuation's secondsimulate()lets the interrupt escape the test and pytest stopsSo every fix is pinned by at least one test that fails without it, and nothing unrelated breaks; the first and last rows flip more than one test because the other serial tests genuinely depend on those two fixes, not because the tests are loose. With the change:
What was not verified
tests/integration/simulation/test_monte_carlo.py::test_monte_carlo_simulate[True]did not finish here, on this branch or without it. Same command and the same 240-second limit on both:The behavior is identical with and without the change, so this is the pre-existing local hang noted in #709 rather than something introduced here. The other ten cases in that file pass with
--runslowin 54.56s. The test carries@pytest.mark.slow, so it runs in the scheduledtest-pytest-slow.yamljob rather than in the pull request gate; a maintainer running that job will get the result I could not.Relationship to #1054
#1054 rewrites both handlers and records the interrupt as
self._interrupted, but it reads that flag only in__check_each_index_was_recorded_once, where it suppresses the missing-simulations check. That is correct for what #1054 is about — an interrupted run has gaps in its indices and must not be reported as corrupt — but it does not re-raise, so #1151 survives it. The two changes are independent; whichever lands second is a small rebase. #1054 fixes theinputs_jsonbinding the same way, as the first statement in the loop, and its_bring_the_fleet_downis where a bounded shutdown belongs.No CHANGELOG entry, following the convention every pull request merged this month has used:
changelog.ymlwrites the entry on merge and contributors leave the file alone. #1173 reports that the workflow has not run since #1112, so this one will need whatever backfill that issue settles on rather than a hand-written line here.Measured on Python 3.12.3, Linux.