Skip to content

[PT-D][Checkpoint]Add MultiThreaded FileSystemWriter for distributed checkpointing and Update tests #87652

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wz337 wants to merge 0 commits into from
Closed

[PT-D][Checkpoint]Add MultiThreaded FileSystemWriter for distributed checkpointing and Update tests #87652

wz337 wants to merge 0 commits into from

Conversation

@wz337
Copy link
Contributor

@wz337 wz337 commented Oct 24, 2022 •
edited by pytorch-bot bot
Loading

This PR includes:

  1. Changes from @kumpera (MultiThreaded FileSystemWriter for distributed checkpointing. #86327): adding MultiThreaded FileSystemWriter for distributed checkpointing, which adds two knobs to FileSystemWriter: thread_count and per_thread_copy_ahead. This increases up to 50% performance improvement on 32 GPUS workloads on AWS.
  2. Add parametrize tests to /test/distributed/_shard/checkpoint/test_file_system_checkpoint.py and /test/distributed/_shard/checkpoint/test_file_system_checkpoint_cpu.py
  3. Modify @with_comms in ShardedTensorTestBase to take in *args and **kwargs.

Tests:

python3 test/distributed/_shard/checkpoint/test_file_system_checkpoint.py
python3 test/distributed/_shard/checkpoint/test_file_system_checkpoint_cpu.py

[T134844615]

cc @jansel @mlazos @soumith @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @chunyuan-w @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx

@pytorch-bot
Copy link

pytorch-bot bot commented Oct 24, 2022 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/87652

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Failures

As of commit 2e48b47:

The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added the release notes: distributed (sharded) release notes category label Oct 24, 2022
@wz337 wz337 requested review from fduwjj and wanchaol October 24, 2022 23:32
@wz337 wz337 changed the title Add tests to fast writer Add MultiThreaded FileSystemWriter for distributed checkpointing and Update tests Oct 24, 2022
@wz337 wz337 changed the title Add MultiThreaded FileSystemWriter for distributed checkpointing and Update tests [WIP]Add MultiThreaded FileSystemWriter for distributed checkpointing and Update tests Oct 25, 2022
@wz337 wz337 marked this pull request as draft October 25, 2022 23:46
@wz337 wz337 marked this pull request as ready for review October 28, 2022 03:41
@pytorch-bot pytorch-bot bot added the ciflow/mps Run MPS tests (subset of trunk) label Oct 28, 2022
@wz337 wz337 changed the title [WIP]Add MultiThreaded FileSystemWriter for distributed checkpointing and Update tests [PT-D][Checkpointing]Add MultiThreaded FileSystemWriter for distributed checkpointing and Update tests Oct 28, 2022
@wz337 wz337 marked this pull request as draft October 28, 2022 04:37
@wz337 wz337 closed this Oct 28, 2022
@wz337 wz337 changed the title [PT-D][Checkpointing]Add MultiThreaded FileSystemWriter for distributed checkpointing and Update tests [PT-D][Checkpoint]Add MultiThreaded FileSystemWriter for distributed checkpointing and Update tests Nov 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mrshenli mrshenli Awaiting requested review from mrshenli

@pritamdamania87 pritamdamania87 Awaiting requested review from pritamdamania87

@zhaojuanmao zhaojuanmao Awaiting requested review from zhaojuanmao

@rohan-varma rohan-varma Awaiting requested review from rohan-varma

@H-Huang H-Huang Awaiting requested review from H-Huang

@awgu awgu Awaiting requested review from awgu

@kwen2501 kwen2501 Awaiting requested review from kwen2501

@fduwjj fduwjj Awaiting requested review from fduwjj

@wanchaol wanchaol Awaiting requested review from wanchaol

@ezyang ezyang Awaiting requested review from ezyang

@Chillee Chillee Awaiting requested review from Chillee

@robieta robieta Awaiting requested review from robieta

@kulinseth kulinseth Awaiting requested review from kulinseth

@mruberry mruberry Awaiting requested review from mruberry

@ngimel ngimel Awaiting requested review from ngimel

@lezcano lezcano Awaiting requested review from lezcano

@nikitaved nikitaved Awaiting requested review from nikitaved

@IvanYashchuk IvanYashchuk Awaiting requested review from IvanYashchuk

@jeffdaily jeffdaily Awaiting requested review from jeffdaily

@BowenBao BowenBao Awaiting requested review from BowenBao

@abock abock Awaiting requested review from abock

@jerryzh168 jerryzh168 Awaiting requested review from jerryzh168

@z-a-f z-a-f Awaiting requested review from z-a-f

@HDCharles HDCharles Awaiting requested review from HDCharles

@salilsdesai salilsdesai Awaiting requested review from salilsdesai

@kimishpatel kimishpatel Awaiting requested review from kimishpatel

@digantdesai digantdesai Awaiting requested review from digantdesai

@jianyuh jianyuh Awaiting requested review from jianyuh

@albanD albanD Awaiting requested review from albanD

@jbschlosser jbschlosser Awaiting requested review from jbschlosser

@soulitzer soulitzer Awaiting requested review from soulitzer

Assignees

No one assigned

Labels

ciflow/mps Run MPS tests (subset of trunk) release notes: distributed (sharded) release notes category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@wz337