Skip to content

CI - add Windows A100 runner - #62

Merged
Alok-Joshi merged 40 commits into
mainfrom
aljoshi/ci-add-windows-a100
Oct 9, 2026
Merged

Alok-Joshi merged 40 commits into
mainfrom
aljoshi/ci-add-windows-a100

Conversation

@Alok-Joshi

@Alok-Joshi Alok-Joshi commented Jul 28, 2026 •

Copy link
Copy Markdown
Collaborator

Add a Windows A100 runner to CI as a new test-ncu-windows job, alongside the Linux test-ncu and test-cupti jobs.

The Windows job installs Nsight Compute 2026.3.0 (CUDA 13.4) and the latest MSVC v14 redistributable (torch's DLLs need it on a fresh image) on the runner.

It installs the project with pip install -e ".[cu13]" plus pytest, and the test dependencies torch (CUDA 13.2 wheels) and cuda-tile. The cupti extra is not installed, since the CUPTI backend is not supported on Windows.

Tests run with MPLBACKEND=Agg, since matplotlib defaults to TkAgg on Windows and tkinter is not properly configured on the runner.

WAR for a known NCU issue on Windows: after pytest finishes, the Python process aborts at exit with 0xC0000409 (FAIL_FAST_FATAL_APP_EXIT), so the step fails even when every test passed. The profiling results are correct; only process exit is affected.

  • pytest runs through a small python -c wrapper that writes pytest's own exit code to pytest_rc.txt before the process exits.
  • If the process exits with 0xC0000409 and that file exists, the step reports pytest's exit code instead. Any other exit code is reported as is, so real test failures still fail the job.
  • The step also prints the raw exit code received from python, in decimal and hex.

The Windows job does not use hatch: hatch run turns the 0xC0000409 exit code into -1 when it passes it on, so the WAR could not detect it.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Alok Joshi <aljoshi@nvidia.com>
@Alok-Joshi Alok-Joshi self-assigned this Jul 28, 2026
@copy-pr-bot

copy-pr-bot Bot commented Jul 28, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@Alok-Joshi

Copy link
Copy Markdown
Collaborator Author

/ok to test 69d5b61

@Alok-Joshi
Alok-Joshi marked this pull request as draft July 28, 2026 06:26
Alok-Joshi and others added 2 commits July 29, 2026 11:05
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Alok Joshi <aljoshi@nvidia.com>
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Alok Joshi <aljoshi@nvidia.com>
@Alok-Joshi

Copy link
Copy Markdown
Collaborator Author

/ok to test 9644f81

Signed-off-by: Alok Joshi <aljoshi@nvidia.com>
@Alok-Joshi

Copy link
Copy Markdown
Collaborator Author

/ok to test bbd6b37

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Alok Joshi <aljoshi@nvidia.com>
@Alok-Joshi

Copy link
Copy Markdown
Collaborator Author

/ok to test 74200ce

The Windows runner is a fresh Windows Server 2025 image with no Visual
C++ Redistributable installed, so torch's c10.dll fails to initialize
(WinError 1114) and every test module importing torch errors during
collection. Install the 2015-2022 x64 redistributable before installing
torch, and verify msvcp140.dll and torch load before running the suite.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Alok Joshi <aljoshi@nvidia.com>
@Alok-Joshi

Copy link
Copy Markdown
Collaborator Author

/ok to test 5bf3a6f

examples/12_cutile.py imports cuda.tile at module scope, so test_12_cutile
fails without it. Install cuda-tile[tileiras] into the same interpreter as
torch, which the hatch test env picks up via system-packages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Alok Joshi <aljoshi@nvidia.com>
@Alok-Joshi

Copy link
Copy Markdown
Collaborator Author

/ok to test 6dbf401

cuda.tile locates the pip-installed tileiras under nvidia.cu13.__path__[0],
which is a namespace package that can span several site-packages. Print it
from the hatch test_cu13 env to see which location the tests actually search.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Alok Joshi <aljoshi@nvidia.com>
@Alok-Joshi

Copy link
Copy Markdown
Collaborator Author

/ok to test 7ed6ffa

Installing torch and cuda-tile into the interpreter that hatch inherits left
them in a different site-packages from the one the tests run in. cuda.tile
resolves its compiler under nvidia.cu13.__path__[0], which is the hatch env,
so the tileiras installed alongside the base interpreter was never found.
Install both through hatch run test_cu13:pip, and verify against that env.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Alok Joshi <aljoshi@nvidia.com>
@Alok-Joshi

Copy link
Copy Markdown
Collaborator Author

/ok to test adadc13

@Alok-Joshi

Copy link
Copy Markdown
Collaborator Author

/ok to test cc24732

@Alok-Joshi

Copy link
Copy Markdown
Collaborator Author

/ok to test d72b30d

Alok-Joshi and others added 2 commits September 11, 2026 17:16
Drop the matplotlib backend probes and the environment introspection added
while diagnosing the Windows job.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Alok Joshi <aljoshi@nvidia.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Alok Joshi <aljoshi@nvidia.com>
@Alok-Joshi

Copy link
Copy Markdown
Collaborator Author

/ok to test 607031d

@Alok-Joshi

Copy link
Copy Markdown
Collaborator Author

/ok to test 4360a1e

@Alok-Joshi

Copy link
Copy Markdown
Collaborator Author

/ok to test cb7585c

hatch turns the 0xC0000409 exit code into -1 when it passes it on, so
the exit-code WAR in the Windows test step never matched. Install the
project, PyTorch and cuTile with pip and run the tests with plain
python instead. This also drops the Windows feature override for the
test_cu13 hatch environment, which is no longer used.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Alok Joshi <aljoshi@nvidia.com>
@Alok-Joshi

Copy link
Copy Markdown
Collaborator Author

/ok to test f42c254

Alok-Joshi and others added 2 commits October 8, 2026 13:43
GitHub labels a step's log with the first line of its script, so the
comment showed up as "Run # WAR for a known NCU issue ...". Keep it as a
YAML comment above run: so the log shows the pytest command instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Alok Joshi <aljoshi@nvidia.com>
Log the exit code PowerShell receives from python, in decimal and hex,
before the WAR adjusts it, to make the WAR's behaviour visible in CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Alok Joshi <aljoshi@nvidia.com>
@Alok-Joshi

Copy link
Copy Markdown
Collaborator Author

/ok to test a3ec5e6

@Alok-Joshi
Alok-Joshi force-pushed the aljoshi/ci-add-windows-a100 branch from a3ec5e6 to def2b9a Compare October 8, 2026 08:37
@Alok-Joshi

Copy link
Copy Markdown
Collaborator Author

/ok to test def2b9a

@Alok-Joshi
Alok-Joshi marked this pull request as ready for review October 8, 2026 08:52
Comment thread .github/workflows/ci.yaml Resolved
Comment thread .github/workflows/ci.yaml Resolved
Comment thread .github/workflows/ci.yaml Resolved
Comment thread .github/workflows/ci.yaml Resolved
Without -f, curl saves an HTTP error page as the installer and exits 0.
Use -f so curl fails on HTTP errors, and check its exit code right away:
in a pwsh step only the last program's exit code fails the step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Alok Joshi <aljoshi@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 9, 2026

Copy link
Copy Markdown

/ok to test 98ec6d9

@Alok-Joshi, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

@Alok-Joshi

Copy link
Copy Markdown
Collaborator Author

/ok to test 98ec6d9

Use the test_cu13 hatch environment in the Windows job again, with a
Windows override that drops the cupti extra, since the CUPTI backend is
not supported on Windows. Install PyTorch and cuTile into it with hatch.

Run pytest through `hatch run` and exit with os._exit after flushing
the output, to try to skip the Python shutdown in which the process
aborts with 0xC0000409 and report pytest's own exit code. This replaces
the exit-code WAR in the test step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Alok Joshi <aljoshi@nvidia.com>
@Alok-Joshi

Copy link
Copy Markdown
Collaborator Author

/ok to test 91c0d29

Activate the test_cu13 hatch environment and run python directly rather
than through `hatch run`, which turns the 0xC0000409 exit code into -1,
and print the exit code to show whether os._exit avoids the abort.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Alok Joshi <aljoshi@nvidia.com>
@Alok-Joshi

Copy link
Copy Markdown
Collaborator Author

/ok to test 4703d70

@Alok-Joshi

Copy link
Copy Markdown
Collaborator Author

/ok to test cb6057e

@Alok-Joshi
Alok-Joshi force-pushed the aljoshi/ci-add-windows-a100 branch from cb6057e to 5363a1e Compare October 9, 2026 12:33
os._exit does not avoid the 0xC0000409 exit code. Go back to saving
pytest's exit code to pytest_rc.txt and reporting it when python exits
with 0xC0000409.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Alok Joshi <aljoshi@nvidia.com>
@Alok-Joshi
Alok-Joshi force-pushed the aljoshi/ci-add-windows-a100 branch from 5363a1e to 79696c8 Compare October 9, 2026 12:37
@Alok-Joshi

Copy link
Copy Markdown
Collaborator Author

/ok to test 79696c8

@Alok-Joshi
Alok-Joshi merged commit 3fbcd0e into main Oct 9, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants