📚 Documentation

The documentation is here: https://pytorch.org/docs/stable/notes/amp_examples.html#dataparallel-in-a-single-process
and here: https://github.com/pytorch/pytorch/blob/master/docs/source/notes/amp_examples.rst#dataparallel-in-a-single-process

This is the statement in question:

torch.nn.DataParallel spawns threads to run the forward pass on each device. The autocast state is thread local, so the following will not work:

This pull request updated parallel_apply so that nn.DataParallel will work with autocast. #43102