[FSDP] Fix clip_grad_norm_() for low prec grads
#90028
Conversation
[ghstack-poisoned]
awgu
requested review from
H-Huang,
kwen2501,
mrshenli,
pritamdamania87,
rohan-varma and
zhaojuanmao
as code owners
pytorch-bot
bot
added
the
release notes: distributed (fsdp)
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/90028
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit d9fd5d8: This comment was automatically generated by Dr. CI and updates every 15 minutes. |
[ghstack-poisoned]
For PyTorch FSDP, the only way that gradients are in low precision is if `keep_low_precision_grads=True` or if the user turns on AMP. This PR adds tests for the former and improves the documentation for `clip_grad_norm_()`, especially around these non-full-precision cases. [ghstack-poisoned]
rohan-varma
left a comment
rohan-varma
left a comment
There was a problem hiding this comment.
Great catch! Do we know why this was not picked up by unittests earlier?
| applied per subset of model parameters. | ||
| .. note:: If every FSDP instance uses ``NO_SHARD``, meaning that no | ||
| gradients are sharded across ranks, then you may directly use | ||
| :func:`torch.nn.utils.clip_grad_norm_`. |
There was a problem hiding this comment.
Can we warn explicitly about this?
There was a problem hiding this comment.
I have it so that if all instances use NO_SHARD, then this method returns torch.nn.utils.clip_grad_norm_(), so it is equivalent.
For PyTorch FSDP, the only way that gradients are in low precision is if `keep_low_precision_grads=True` or if the user turns on AMP. This PR adds tests for the former and improves the documentation for `clip_grad_norm_()`, especially around these non-full-precision cases. [ghstack-poisoned]
We did not not test |
For PyTorch FSDP, the only way that gradients are in low precision is if `keep_low_precision_grads=True` or if the user turns on AMP. This PR adds tests for the former and improves the documentation for `clip_grad_norm_()`, especially around these non-full-precision cases. [ghstack-poisoned]
ghstack-source-id: 5ae7c09 Pull Request resolved: pytorch#90028
For PyTorch FSDP, the only way that gradients are in low precision is if `keep_low_precision_grads=True` or if the user turns on AMP. This PR adds tests for the former and improves the documentation for `clip_grad_norm_()`, especially around these non-full-precision cases. [ghstack-poisoned]
|
@pytorchbot rebase -s |
|
@pytorchbot successfully started a rebase job. Check the current status here |
For PyTorch FSDP, the only way that gradients are in low precision is if `keep_low_precision_grads=True` or if the user turns on AMP. This PR adds tests for the former and improves the documentation for `clip_grad_norm_()`, especially around these non-full-precision cases. [ghstack-poisoned]
|
Successfully rebased |
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
For PyTorch FSDP, the only way that gradients are in low precision is if `keep_low_precision_grads=True` or if the user turns on AMP. This PR adds tests for the former and improves the documentation for `clip_grad_norm_()`, especially around these non-full-precision cases. Pull Request resolved: pytorch#90028 Approved by: https://github.com/rohan-varma
Stack from ghstack (oldest at bottom):
clip_grad_norm_()for low prec grads #90028keep_low_precision_grads=Trueforuse_orig_params=True#90027For PyTorch FSDP, the only way that gradients are in low precision is if
keep_low_precision_grads=Trueor if the user turns on AMP. This PR adds tests for the former and improves the documentation forclip_grad_norm_(), especially around these non-full-precision cases.