Naturally this particular CI run is green. :)

But the cause is unrelated to the diff, with thousands of tests I've managed to
reproduce it once now. Still trying to isolate it.

@kostmo can we get a CI scan for the nuclear_norm failure?

The latest status is unfortunately "Heisenbug": If I repeat the tests often enough (1000+), print out the input values on failure, then run the test manually with the inputs, the results are the same.

This is one example (test_torch.py is modified for printing):

test_torch.py:1075: in _test_nuclear_norm_axes
    check_single_nuclear_norm(rr, x, axes)
test_torch.py:1001: in check_single_nuclear_norm
    self.assertTrue(np.allclose(ans.cpu(), expected, rtol=1e-02, atol=1e-03, equal_nan=True), msg=msg)
E   AssertionError: False is not true : r: tensor([[[[-0.47391583903785255982,  0.99425932417625995097]],
E   
E            [[-0.16560788865645595380, -0.94544471859987488926]]],
E   
E   
E           [[[ 0.86797840599953324237, -0.07809049467944455258]],
E   
E            [[-1.15369110156186338578, -0.83403423341840843275]]]],
E          device='cuda:0')
E   x: tensor([[[[-0.47391583903785255982]],
E   
E            [[-0.16560788865645595380]]],
E   
E   
E           [[[ 0.86797840599953324237]],
E   
E            [[-1.15369110156186338578]]]], device='cuda:0')
E   axes: (1, 3)
E   ans: tensor([[0.50201812245794563694],
E           [0.52862857287650466542]], device='cuda:0')
E   expected: array([[0.50201812],
E          [1.44374155]])
E   strides: (16, 8, 8, 8)

Equivalent to:

import torch
import numpy as np


axes = (1, 3)

r = torch.tensor([[[[-0.47391583903785255982,  0.99425932417625995097]],
                   [[-0.16560788865645595380, -0.94544471859987488926]]],
                  [[[ 0.86797840599953324237, -0.07809049467944455258]],
                   [[-1.15369110156186338578, -0.83403423341840843275]]]],
                 dtype=torch.float32, device='cuda:0')

x = r[:, :, :, ::2]

a = np.array(x.cpu(), copy=False)

expected = np.linalg.norm(a, "nuc", axis=axes)

ans = torch.norm(x, "nuc", dim=axes)

print("%r\n%r\n" % (ans, expected))

Which gives:

tensor([[0.5020],
        [1.4437]], device='cuda:0')
array([[0.50201815],
       [1.4437416 ]], dtype=float32)

Can you run the kernel with cuda-memcheck? Usually, if it doesn't reproduce when you rerun the inputs, it's because you're accessing uninitialized memory of some sort or another. A recent similar bug we fixed was #21392

@ezyang I tried cuda-memcheck, but it always showed 0 errors, even when the tests fail. It did seem to accelerate the test failures though (no repetitions needed).

So the situation is:

1. In the unmodified master, cuda-memcheck accelerates the test failure but shows 0 errors.

2. When extracting all tests from unittest into a separate python script, the tests always pass.

3. When moving the tests from test_cuda to test_torch, the tests pass, also with cuda_memcheck.

So 3) is what the latest diff implements. It is not quite satisfying, but may help in suppressing the flakiness until the cause is found.

I haven't tried --tool initcheck yet though. Still there seems to be some interaction with other tests.

@vishwakftw Thanks, I think @skipCUDANonDefaultStreamIf(True) solves the problem. At least I can no longer observe the behavior that cuda-memcheck reliably triggers the issue instantly.

cuda-memcheck --tool initcheck still shows uninitialized read accesses in magma, but that also occurs in other tests (svd for instance) and may be just a harmless magma issue.

@ezyang I think the latest diff (suggestion by @vishwakftw) resolves #21785.

@ezyang merged this pull request in 7108218.