Skip to content

Enable channels_last_3d on SyncBatchNorm #88401

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
xwang233 wants to merge 8 commits into from
Closed

Enable channels_last_3d on SyncBatchNorm #88401

xwang233 wants to merge 8 commits into from

Conversation

@xwang233
Copy link
Collaborator

@xwang233 xwang233 commented Nov 3, 2022 •
edited
Loading

This PR enabled the use of fast channels_last kernels on SyncBatchNorm with channels_last_3d memory format.

With a small benchmark script here #88021 (comment), on V100, I got

master:

DDP channels_last=False, run_forward_backward, time: 0.8945400714874268 sec
DDP channels_last=True, run_forward_backward, time: 1.4736433029174805 sec

This PR:

DDP channels_last=False, run_forward_backward, time: 0.8927242755889893 sec
DDP channels_last=True, run_forward_backward, time: 0.48697471618652344 sec

This PR is a follow-up of #46906

Close #88021

@xwang233 xwang233 requested review from ngimel and ptrblck November 3, 2022 04:49
@pytorch-bot pytorch-bot bot added the release notes: nn release notes category label Nov 3, 2022
@pytorch-bot
Copy link

pytorch-bot bot commented Nov 3, 2022 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/88401

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Failures

As of commit 5b2ce61:

The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Nov 3, 2022 •
edited
Loading

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: xwang233 / name: Wang, Xiao (f5fdfa2)

@zou3519 zou3519 added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Nov 3, 2022
@myron
Copy link

myron commented Nov 3, 2022

@xwang233 Looks great, thank you so much for adding it!

@xwang233
Copy link
Collaborator Author

xwang233 commented Nov 11, 2022

Hi @ngimel , would you take a look at this PR? All tests are passed. Thanks!

ngimel
ngimel reviewed Nov 11, 2022
View reviewed changes
Copy link
Collaborator

@ngimel ngimel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor testing suggestion, otherwise looks good, thanks for the fix!

torch/testing/_internal/distributed/distributed_test.py Outdated
model_gpu, device_ids=[rank]
)

memory_format = torch.channels_last_3d
Copy link
Collaborator

@ngimel ngimel Nov 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like preparing inputs is the only difference with channels last 2d, can you refactor everything else into a helper function for these 2 tests?

@xwang233
Copy link
Collaborator Author

xwang233 commented Nov 15, 2022

Hi @ngimel , I've merged the test code with channels_last and channels_last_3d into one test. All the CI checks look green now.

ngimel
ngimel approved these changes Nov 15, 2022
View reviewed changes
Copy link
Collaborator

@ngimel ngimel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thanks!

@ngimel
Copy link
Collaborator

ngimel commented Nov 15, 2022

@pytorchbot merge -g

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Nov 15, 2022
@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Nov 15, 2022

Merge started

Your change will be merged once all checks on your PR pass since you used the green (-g) flag (ETA: 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Nov 15, 2022

Merge failed

Reason: 2 additional jobs have failed, first few of them are: trunk ,trunk / cuda11.6-py3.10-gcc7-sm86 / test (default, 3, 4, linux.g5.4xlarge.nvidia.gpu)

Details for Dev Infra team Raised by workflow job

@xwang233
Copy link
Collaborator Author

xwang233 commented Nov 15, 2022

@pytorchmergebot merge -f "failures are irrelevant"

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Nov 15, 2022

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

kulinseth pushed a commit to kulinseth/pytorch that referenced this pull request Dec 10, 2022
@xwang233 @kulinseth
Enable channels_last_3d on SyncBatchNorm (pytorch#88401)
b2b14e7 
This PR enabled the use of fast channels_last kernels on SyncBatchNorm with channels_last_3d memory format.

With a small benchmark script here pytorch#88021 (comment), on V100, I got

master:
```
DDP channels_last=False, run_forward_backward, time: 0.8945400714874268 sec
DDP channels_last=True, run_forward_backward, time: 1.4736433029174805 sec
```

This PR:
```
DDP channels_last=False, run_forward_backward, time: 0.8927242755889893 sec
DDP channels_last=True, run_forward_backward, time: 0.48697471618652344 sec
```

This PR is a follow-up of pytorch#46906

Close pytorch#88021
Pull Request resolved: pytorch#88401
Approved by: https://github.com/ngimel
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ngimel ngimel ngimel approved these changes

@ptrblck ptrblck Awaiting requested review from ptrblck

@mrshenli mrshenli Awaiting requested review from mrshenli

@pritamdamania87 pritamdamania87 Awaiting requested review from pritamdamania87

@zhaojuanmao zhaojuanmao Awaiting requested review from zhaojuanmao

@rohan-varma rohan-varma Awaiting requested review from rohan-varma

@H-Huang H-Huang Awaiting requested review from H-Huang

@awgu awgu Awaiting requested review from awgu

@kwen2501 kwen2501 Awaiting requested review from kwen2501

@albanD albanD Awaiting requested review from albanD

@jbschlosser jbschlosser Awaiting requested review from jbschlosser

Assignees

No one assigned

Labels

ciflow/trunk Trigger trunk jobs on your pull request Merged open source release notes: nn release notes category triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

SyncBatchNorm 3D is slow with torch.channels_last_3d format in DDP on GPU

6 participants

@xwang233 @myron @ngimel @pytorchmergebot @zou3519 @pytorchbot