Skip to content

derive_metric results are wrong, or aggregation fails, for runs that failed under ignore_failures=True #89

Closed

Description

@0z5a

Description

When a run fails inside nsight.annotate(..., ignore_failures=True), derive_metric is not called for it (there are no values). The derived row of that run is then recorded as if derive_metric had returned a bare number: the metric is named after the function, the unit is NaN, and a "does not have a unit specified" warning is emitted.

Case A, derive_metric returns a dict and one configuration always fails:

import nsight
import torch

def per_element(time_ns, n):
    return {
        "elements_per_ns": (n * n / time_ns, "elem/ns"),
        "ns_per_element": (time_ns / (n * n), "ns/elem"),
    }

@nsight.analyze.kernel(configs=[(64,), (128,)], runs=2, derive_metric=per_element)
def bench(n):
    a = torch.randn(n, n, device="cuda")
    with nsight.annotate("add", ignore_failures=True):
        if n == 128:
            raise RuntimeError("this config is not supported")
        _ = a + a

print(bench().to_dataframe()[["Annotation", "n", "Metric", "NumRuns", "Unit"]])
UserWarning: Derived metric 'per_element' does not have a unit specified. Return a tuple (value, unit) instead of just the value. ...
Annotation   n                 Metric  NumRuns    Unit
       add  64 gpu__time_duration.sum        2      ns
       add 128 gpu__time_duration.sum        0      ns
       add  64        elements_per_ns        2 elem/ns
       add  64         ns_per_element        2 ns/elem
       add 128            per_element        0     NaN

n=128 has a row for a metric called per_element, and no rows for elements_per_ns / ns_per_element.

Case B, derive_metric returns (value, unit) and only one of several runs of a configuration fails (runs=3, the second run raises):

AssertionError: Column 'Unit' has multiple values in group: <StringArray>
['elem/ns', nan]

The call fails in aggregate_data and returns nothing, although two runs succeeded.

Expected

The failed run counts as a missing value of the same derived metrics, with their units, the way gpu__time_duration.sum is already reported for the failed configuration above. No unit warning when derive_metric returns units. The ignore_failures docstring says "The measured metric for this run will be set to NaN".

Where

extract_df_from_data in nsight/extraction.py: derived_metric is None for a failed run and falls into the branch for a scalar return value. In case B the Unit column then has two values in one group, which the aggregation in nsight/transformation.py rejects.

Environment

nsight-python main at c89d4b5, Python 3.12.3, pandas 3.1.1, Linux x86_64.

How this was checked: no GPU was available, so the outputs above come from the same decorator and annotation calls with the profiling backend replaced by a mock. The mock reports the kernel as fake_kernel, and that column is cut from the table above. The decorator, run loop, extraction and aggregation were the real nsight code. The claim that the real NCU and CUPTI collectors pass no values for a failed run is based on reading extraction.py and collection/cupti.py, not on a GPU run.

Draft PR #74 changes the same function but not this part. A fix is ready if you approve it: it gives failed-run rows the metric names and units of a successful run, and adds two unit tests in tests/test_collection.py. One open point is what to report when every run of a configuration fails, because then there is no successful run to copy names and units from. The fix keeps today's output for that case, without the false warning.

This was found with an AI coding assistant (Claude Code, claude-fable-5-2), which also wrote and ran the mocked repro and drafted this report.

Activity

  1. 0z5a commented on Nov 11, 2026

    @0z5a
    ContributorAuthor

    A fix with a regression test is ready for this. Could this issue be assigned to me? I can then open a pull request.

  2. added a commit that references this issue on Nov 17, 2026
  3. added a commit that references this issue on Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions