🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/165971

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 4d17f59 with merge base 03f3f78 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

While it is too expensive to create large dense tensors, is it possible to create large sparse tensors to test the ops?

Of course!

import torch
from torch.optim import Adagrad


def test_torch_adagrad():
    num_params = 27000008
    param_size = 192
    param = torch.randn(num_params, param_size, device="cuda", dtype=torch.float32, requires_grad=True)
    grad = torch.randn_like(param) * 0.01
    param.grad = grad
    optimizer = Adagrad([param], lr=0.01)
    optimizer.step()
    torch.cuda.synchronize()


if __name__ == "__main__":
    test_torch_adagrad()

I am so sorry. I cannot run the code,could you run it? @cyyever

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 19.31 GiB. GPU 0 has a total capacity of 31.74 GiB of which 12.12 GiB is free. Including non-PyTorch memory, this process has 19.61 GiB memory in use. Of the allocated memory 19.31 GiB is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

@lingebeng

import torch
from torch.optim import Adagrad


def test_torch_adagrad():
    num_params = 27000008
    param_size = 150
    param = torch.randn(num_params, param_size, device="cuda:1", dtype=torch.bfloat16, requires_grad=True)
    grad = torch.randn_like(param) * 0.01
    param.grad = grad
    optimizer = Adagrad([param], lr=0.01)
    optimizer.step()
    torch.cuda.synchronize()


if __name__ == "__main__":
    test_torch_adagrad()

This one allocates less than 40GB, but no error raises.

Thanks,I see. Maybe it's not a bug!

@lingebeng We can have further chats via e-mail.

OK,I have contacted you!

@pytorchbot merge

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

@albanD Yes, MSVC still recognises int as int32_t, likely for Win32 compatibility.