Skip to content

#89 - Keep derived metric names and units for failed runs - #92

Merged
Alok-Joshi merged 2 commits into
NVIDIA:mainfrom
0z5a:fix/derive-metric-failed-runs
Nov 18, 2026
Merged

Alok-Joshi merged 2 commits into
NVIDIA:mainfrom
0z5a:fix/derive-metric-failed-runs

Conversation

@0z5a

@0z5a 0z5a commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Fixes #89

A run that fails inside nsight.annotate(..., ignore_failures=True) has no metric values, so derive_metric is not called for it. Its derived row was recorded under the name of the derive_metric function with no unit, and a warning asked for a (value, unit) return value even when derive_metric returns one.

  • With a dict-returning derive_metric, the failed configuration got a metric named after the function instead of NaN rows for the dict keys.
  • With a (value, unit) return value, a configuration in which only some runs failed made aggregate_data raise Column 'Unit' has multiple values in group.

Change

  • nsight/extraction.py: the derived metrics of a failed run are recorded as missing values under the names and units returned for a successful run. The missing-unit warning is only emitted when derive_metric was called. If every run failed, the output is unchanged apart from the warning.
  • tests/test_collection.py: test_extract_df_from_data_failed_run_keeps_derived_metric_unit and test_extract_df_from_data_failed_run_keeps_derived_metric_names.

Names and units are taken from the first successful row, which assumes derive_metric returns the same keys and units on every call.

Testing

Run on x86_64 Linux without a GPU (CUDA_VISIBLE_DEVICES empty), Python 3.12.3, pandas 3.1.1, on main at fcb47f2:

  • pytest tests/test_collection.py: main with only the new tests added, 2 failed, 18 passed; with this change, 20 passed.
  • pytest tests/test_collection.py tests/test_transformation.py tests/test_info_collector.py: 54 passed on main, 56 passed with this change (the two new tests).
  • black --check ., isort --check ., mypy --strict --ignore-missing-imports . (lint environment with the project, black, isort and mypy): clean.

Not run: the GPU tests and the examples (NCU and CUPTI). The new tests build the per-annotation data by hand, as the other tests in tests/test_collection.py do; there is no GPU test for this path yet. The Nsight Compute on the machine used was older than the version the tests require.

No release-note entry is included; I can add one to this PR if you want it.

This change was drafted with Claude Code (claude-fable-5-2, rebase and PR text with claude-opus-5-5); the test results above are from the runs described.

A run that fails inside nsight.annotate(..., ignore_failures=True) has
no metric values, so derive_metric is not called for it. Its derived
row was recorded under the name of the derive_metric function with no
unit, and a warning asked for a (value, unit) return value even when
derive_metric returns one.

With a dict-returning derive_metric this added a metric named after the
function instead of NaN rows for the dict keys. With a (value, unit)
return value, a configuration in which only some runs failed made
aggregate_data raise "Column 'Unit' has multiple values in group".

Record the derived metrics of a failed run as missing values under the
names and units returned for a successful run, and warn about a missing
unit only when derive_metric was called.

Signed-off-by: Dezhen Lu <dezhen.lu@student.uni-tuebingen.de>
@copy-pr-bot

copy-pr-bot Bot commented Nov 13, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@nvssatoor
nvssatoor requested a review from Alok-Joshi November 16, 2026 06:21
@Alok-Joshi

Copy link
Copy Markdown
Collaborator

/ok to test 415b25c

@Alok-Joshi

Alok-Joshi commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

It would be good to add a test parallel to tests/test_profiler.py::test_parameter_derive_metric which tries out all possible outputs for a derive_metric function and asserts on the correct dataframe layout. You could call this test test_parameter_derive_metric_with_ignore_failures. The assertion logic in test_parameter_derive_metric may be reused in new test - better to move it in a helper.

You could also cover the case where some runs fail/all runs fail.

@Alok-Joshi

Copy link
Copy Markdown
Collaborator

Please document this behaviour in the derive_metric parameter docs in nsight/analyze.py, after the note on units:

Note on failed runs: derive_metric is not called for a run that failed under ignore_failures=True; its derived metrics are NaN, under the metric names and units of a successful run. If no run succeeded, they are labelled with the function name and have no unit.

Add test_parameter_derive_metric_with_ignore_failures, which runs every
derive_metric return shape of test_parameter_derive_metric with
ignore_failures=True, once with the first run of each config failing and
once with every run failing. The dataframe layout checks of
test_parameter_derive_metric move into a helper shared by both tests.

Document in the derive_metric parameter docs that derive_metric is not
called for a failed run and which names and units its derived metrics
get.

Signed-off-by: Dezhen Lu <dezhen.lu@student.uni-tuebingen.de>
@0z5a

0z5a commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review. I added test_parameter_derive_metric_with_ignore_failures, which runs every derive_metric return shape from test_parameter_derive_metric with ignore_failures=True, once with the first run of each config failing and once with all runs failing, and checks the layout, units, NumRuns and warnings; the layout checks now live in a shared helper and the parameter list is shared by both tests. Your note on failed runs is in the derive_metric docs, right after the note on units. I have no GPU where I can collect NCU counters right now, so the new test has not run under NCU or CUPTI yet; on a machine without CUDA (CUDA Python modules stubbed out so the package imports) tests/test_collection.py, tests/test_transformation.py and tests/test_info_collector.py pass (56) and black, isort and mypy --strict are clean. Feeding hand-built run data through extract_df_from_data and aggregate_data and applying the new assertions passes all 18 cases with this PR, while 10 of them fail on main.

@Alok-Joshi

Copy link
Copy Markdown
Collaborator

/ok to test a61ffa9

@Alok-Joshi

Copy link
Copy Markdown
Collaborator

Merging. Thanks @0z5a !

@Alok-Joshi
Alok-Joshi merged commit 18e0836 into NVIDIA:main Nov 18, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

derive_metric results are wrong, or aggregation fails, for runs that failed under ignore_failures=True

2 participants