Repository navigation
#89 - Keep derived metric names and units for failed runs - #92
Conversation
A run that fails inside nsight.annotate(..., ignore_failures=True) has no metric values, so derive_metric is not called for it. Its derived row was recorded under the name of the derive_metric function with no unit, and a warning asked for a (value, unit) return value even when derive_metric returns one. With a dict-returning derive_metric this added a metric named after the function instead of NaN rows for the dict keys. With a (value, unit) return value, a configuration in which only some runs failed made aggregate_data raise "Column 'Unit' has multiple values in group". Record the derived metrics of a failed run as missing values under the names and units returned for a successful run, and warn about a missing unit only when derive_metric was called. Signed-off-by: Dezhen Lu <dezhen.lu@student.uni-tuebingen.de>
|
/ok to test 415b25c |
|
It would be good to add a test parallel to You could also cover the case where some runs fail/all runs fail. |
|
Please document this behaviour in the
|
Add test_parameter_derive_metric_with_ignore_failures, which runs every derive_metric return shape of test_parameter_derive_metric with ignore_failures=True, once with the first run of each config failing and once with every run failing. The dataframe layout checks of test_parameter_derive_metric move into a helper shared by both tests. Document in the derive_metric parameter docs that derive_metric is not called for a failed run and which names and units its derived metrics get. Signed-off-by: Dezhen Lu <dezhen.lu@student.uni-tuebingen.de>
|
Thanks for the review. I added |
|
/ok to test a61ffa9 |
|
Merging. Thanks @0z5a ! |
Fixes #89
A run that fails inside
nsight.annotate(..., ignore_failures=True)has no metric values, soderive_metricis not called for it. Its derived row was recorded under the name of thederive_metricfunction with no unit, and a warning asked for a(value, unit)return value even whenderive_metricreturns one.derive_metric, the failed configuration got a metric named after the function instead of NaN rows for the dict keys.(value, unit)return value, a configuration in which only some runs failed madeaggregate_dataraiseColumn 'Unit' has multiple values in group.Change
nsight/extraction.py: the derived metrics of a failed run are recorded as missing values under the names and units returned for a successful run. The missing-unit warning is only emitted whenderive_metricwas called. If every run failed, the output is unchanged apart from the warning.tests/test_collection.py:test_extract_df_from_data_failed_run_keeps_derived_metric_unitandtest_extract_df_from_data_failed_run_keeps_derived_metric_names.Names and units are taken from the first successful row, which assumes
derive_metricreturns the same keys and units on every call.Testing
Run on x86_64 Linux without a GPU (
CUDA_VISIBLE_DEVICESempty), Python 3.12.3, pandas 3.1.1, onmainatfcb47f2:pytest tests/test_collection.py:mainwith only the new tests added, 2 failed, 18 passed; with this change, 20 passed.pytest tests/test_collection.py tests/test_transformation.py tests/test_info_collector.py: 54 passed onmain, 56 passed with this change (the two new tests).black --check .,isort --check .,mypy --strict --ignore-missing-imports .(lint environment with the project, black, isort and mypy): clean.Not run: the GPU tests and the examples (NCU and CUPTI). The new tests build the per-annotation data by hand, as the other tests in
tests/test_collection.pydo; there is no GPU test for this path yet. The Nsight Compute on the machine used was older than the version the tests require.No release-note entry is included; I can add one to this PR if you want it.
This change was drafted with Claude Code (claude-fable-5-2, rebase and PR text with claude-opus-5-5); the test results above are from the runs described.