Great, very happy to see that this happened. :)

OOC, how did you check that you hit all of the necessary entry points to modify the allocator?

I'll update the PR to provide usage per GPU instead. :)

@ezyang I didn't do any tests, but I checked with @colesbury on that :)
Btw, should the nightly build fail?

Windows failure is the following:

22:45:58 C:\Jenkins\workspace\pytorch-builds\pytorch-win-ws2016-cuda9-cudnn7-py3-build\aten\src\THC\THCCachingAllocator.cpp(172): error C2039: 'max': is not a member of 'std'
22:45:58 C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.11.25503\include\unordered_map(15): note: see declaration of 'std'
22:45:58 C:\Jenkins\workspace\pytorch-builds\pytorch-win-ws2016-cuda9-cudnn7-py3-build\aten\src\THC\THCCachingAllocator.cpp(172): error C3861: 'max': identifier not found

To fix it we can add #include <algorithm>, according to https://social.msdn.microsoft.com/Forums/vstudio/en-US/f5915ad0-a9d1-49f3-8643-ffd623f72b93/error-c2039-max-is-not-a-member-of-std?forum=vcgeneral

@yf225 I'll add it, thanks!

I noticed that you added stats for the "residency" (actual amount of live data), but not memory that was actually allocated from the GPU. The latter is also a pretty useful stat to collect, since (1) it is the thing that end users will actually see when the query nvidia-smi, and (2) it is the thing that will determine if you are actually out of memory or not is the actual memory allocated. Fragmentation could mean we are wasting memory in the caching allocator, that doesn't show up in the residency computation. (Though, tensor allocations tend to be big, so this isn't a big deal in practice...)

There's tons of other stats we could add to the allocator but I guess no one needed them, so we ain't gonna add 'em :)

Re CI failures, I think there is an OOD. Looking.

@pytorchbot retest this please

@ezyang I'd love to add actual allocated GPU usage (unused in cache + context), but I'm not sure how to count the size of the context. Perhaps just getting total cache size is fine for now?

Also, I actually think that total residency memory (i.e. memory pointed by the tensors) is sometimes a better indication for OOM because there can be large amount of available memory in cache. When nvidia-smi shows little memory left, users may still allocate large tensors.

Yep, the number of blocks in the cache is the usual thing to use.

And yes, you're right, residency can be low while the cache is large, which is why residency is important too :)

One way to get more detailed stats would be to just go over all blocks and accumulate information then instead of keeping some running stats. That's more expensive, but I'd assume that these functions are mostly useful for one-off debugging

@apaszke Yes, but the max memory allocated is also a useful stat, so I went with the running stats route.

Yeah, generally you want running stats if at all possible.

Added cache stats. Changed the functions to provide per-device stats.

@pytorchbot retest this please

The build errors are fixed in #4544

@pytorchbot retest this please

again test fails are unrelated, and fixed in #4544