Skip to content

Support variable kernel counts per annotation execution - #74

Open
bastianhagedorn wants to merge 7 commits into
mainfrom
bhagedorn/variable-kernel-counts
Open

bastianhagedorn wants to merge 7 commits into
mainfrom
bhagedorn/variable-kernel-counts

Conversation

@bastianhagedorn

@bastianhagedorn bastianhagedorn commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Configurations that launch different numbers of kernels can currently fail or silently group measurements into the wrong samples. Two and three kernels per configuration raise an error, while two and four can become two three-kernel samples.

Track each annotation execution's configuration and run in a private NVTX label, then reduce only the kernels belonging to that sample. NCU and CUPTI share the grouping path. Kernel counts may vary across configurations and repeated runs when combine_kernel_metrics is supplied.

  • Preserve per-thread annotation-name uniqueness. Different threads using the same public name contribute to one sample for the configuration and run, including across reused worker threads.
  • Retain valid samples when an entered annotation has no profiled kernels. Record a missing sample and warn with the annotation, configuration and run.
  • Report kernel names when a reducer is needed, and validate reducers with a clear combine_kernel_metrics error. Builtins, partials and callable objects remain supported.
  • Keep capture tracking active through collector teardown and preserve main's in-memory custom-info collection, including skipped-run alignment. Reject annotations outside a profiled config/run at annotation entry.
  • Use a capture-local execution counter without a session UUID. Document extraction signatures, missing samples, thread behavior and the saved-report compatibility change in the release notes.

Saved .ncu-rep files now require the matching in-memory invocations mapping for reprocessing. Returned results and exported CSVs remain available for later analysis. The processed Kernel column continues to describe the first sample's kernel sequence.

Validation:

  • 81 CPU tests pass, covering extraction/grouping, custom information, collection and transformation.

  • Six GPU tests pass with NCU 2026.2.1 and the same six pass with CUPTI 13.4 on an NVIDIA A40. Coverage includes 2/3 and 2/4 kernels across configurations, 2 then 4 kernels across repeated runs of one configuration, an empty middle run, annotation-scope collector alignment and the planted-pickle regression.

  • Black, isort and strict mypy pass across all 48 source files in a clean lint environment.

  • Sphinx builds with warnings treated as errors after suppressing a CUDA forward-reference warning also reproduced on unchanged main.

  • Multi-GPU regression: one configuration sweep alternates between two physical A100 80GB PCIe devices. Counts 2/4/3/1, two runs each: all eight raw samples have the expected kernel sequence, device collector value and aggregation. Passes with NCU 2026.2.1 and CUPTI 13.5. The test skips cleanly when fewer than two GPUs are visible.

  • Fresh GitHub CI is authorized and queued for the test-only revision.

Signed-off-by: Bastian Hagedorn <bhagedorn@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 12, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@acollins3

Copy link
Copy Markdown
Collaborator

/ok to test 9c83553

Signed-off-by: Bastian Hagedorn <bhagedorn@nvidia.com>
@acollins3

Copy link
Copy Markdown
Collaborator

/ok to test 34eac68

Signed-off-by: Bastian Hagedorn <bhagedorn@nvidia.com>
@bastianhagedorn

Copy link
Copy Markdown
Collaborator Author

/ok to test 046ae11

@acollins3

Copy link
Copy Markdown
Collaborator

I like this approach. Makes it very clear which kernel belongs to which run.

Before merging, this it conflicts with main, as PR #83 was just merged, so that will need addressed.

Comment thread nsight/_session.py Resolved
Comment thread nsight/_session.py Resolved
Comment thread nsight/extraction.py Resolved
Comment thread nsight/extraction.py Resolved
Comment thread nsight/extraction.py Resolved
Comment thread nsight/extraction.py Resolved
Comment thread nsight/collection/ncu.py Resolved
Comment thread nsight/collection/cupti.py Resolved
Signed-off-by: Bastian Hagedorn <bhagedorn@nvidia.com>
Signed-off-by: Bastian Hagedorn <bhagedorn@nvidia.com>
Comment thread nsight/_session.py Resolved
Comment thread tests/test_profiler.py Resolved

@hsahuja0805 hsahuja0805 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made few comments. Overall, approach looks good to me.
I think this would also cover the scenario when different configuration execute on different GPU's. However, this still needs to be validated.

Signed-off-by: Bastian Hagedorn <bhagedorn@nvidia.com>
@bastianhagedorn

Copy link
Copy Markdown
Collaborator Author

/ok to test 7fb64b0

Signed-off-by: Bastian Hagedorn <bhagedorn@nvidia.com>
@bastianhagedorn

Copy link
Copy Markdown
Collaborator Author

/ok to test d2e9ed8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants