Update torchrun and TorchElastic to take optional `local_addr` param to allow skip local IP lookup if specified #88922

cz-37 · 2022-11-12T01:11:33Z

Summary:
Update dynamic renderzvous nodes to use rendezvous hostname if provided.
For PR: #85300

Before:
For dynamic renderzvous, it always grab the fqdn from socket for each node even if user specified the address.
For example,
https://github.com/pytorch/pytorch/blob/master/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py#L248-L256

return _NodeDesc(socket.getfqdn(), os.getpid(), local_id)

Now:
If user specifies the hostname, each node will respect the given hostname.
For example, socket.getfqdn(<hostname>)

Test Plan: Unit tests.

Differential Revision: D41204028

linux-foundation-easycla · 2022-11-12T01:11:37Z

The committers listed above are authorized under a signed CLA.

✅ login: cz-37 (a544a41a621772cd3bda4cd8fa69d2e73d60189d)

pytorch-bot · 2022-11-12T01:11:37Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/88922

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 92f310e:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2022-11-12T01:12:39Z

This pull request was exported from Phabricator. Differential Revision: D41204028

facebook-github-bot · 2022-11-14T11:01:33Z

This pull request was exported from Phabricator. Differential Revision: D41204028

facebook-github-bot · 2022-11-14T11:34:56Z

This pull request was exported from Phabricator. Differential Revision: D41204028

d4l3k · 2022-11-19T00:30:48Z

torch/distributed/elastic/rendezvous/utils.py

can we add some unit tests for this parsing logic?

d4l3k · 2022-11-19T00:31:50Z

torch/distributed/elastic/rendezvous/dynamic_rendezvous.py

nit:

hostname = param_hostname if param_hostname else ''

or

hostname = param_hostname or ''

d4l3k · 2022-11-19T00:34:30Z

torch/distributed/elastic/rendezvous/dynamic_rendezvous.py

This endpoint could be a different host right?

like if you rendezvous with endpoint == "rank0" on rank 1 machine we wouldn't want to grab rank0 as the current address? am I misunderstanding this?

facebook-github-bot · 2022-11-29T12:43:58Z

This pull request was exported from Phabricator. Differential Revision: D41204028

facebook-github-bot · 2022-12-01T05:02:28Z

This pull request was exported from Phabricator. Differential Revision: D41204028

facebook-github-bot · 2022-12-05T21:43:14Z

This pull request was exported from Phabricator. Differential Revision: D41204028

d4l3k

LGTM see internal comments

facebook-github-bot · 2022-12-19T13:38:41Z

This pull request was exported from Phabricator. Differential Revision: D41204028

facebook-github-bot · 2022-12-20T09:50:11Z

This pull request was exported from Phabricator. Differential Revision: D41204028

…to allow skip local IP lookup if specified (pytorch#88922) Summary: Pull Request resolved: pytorch#88922 Update dynamic renderzvous nodes to use rendezvous hostname if provided. Before: For dynamic renderzvous, it always grab the `fqdn` from socket for each node even if user specified the address. For example, https://github.com/pytorch/pytorch/blob/master/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py#L248-L256 ``` return _NodeDesc(socket.getfqdn(), os.getpid(), local_id) ``` Now: If user specifies the hostname, each node will respect the given hostname. For example, `socket.getfqdn(<hostname>) ` Test Plan: ``` % python -m torch.distributed.run --help usage: run.py [-h] [--nnodes NNODES] [--nproc_per_node NPROC_PER_NODE] [--rdzv_backend RDZV_BACKEND] [--rdzv_endpoint RDZV_ENDPOINT] [--rdzv_id RDZV_ID] [--rdzv_conf RDZV_CONF] [--standalone] [--max_restarts MAX_RESTARTS] [--monitor_interval MONITOR_INTERVAL] [--start_method {spawn,fork,forkserver}] [--role ROLE] [-m] [--no_python] [--run_path] [--log_dir LOG_DIR] [-r % python -m torch.distributed.run --local_addr 192.168.0.84 --rdzv_backend 'c10d' training_script.py NOTE: Redirects are currently not supported in Windows or MacOs. Hello World % python -m torch.distributed.run --rdzv_backend 'c10d' training_script.py NOTE: Redirects are currently not supported in Windows or MacOs. Hello World ``` Unit tests with the 2 added tests. Reviewed By: d4l3k Differential Revision: D41204028 fbshipit-source-id: 6dbf07c10f19c44e691e7dcaa4cb3df62c735a98

facebook-github-bot · 2022-12-20T22:52:32Z

This pull request was exported from Phabricator. Differential Revision: D41204028

facebook-github-bot · 2022-12-21T03:53:14Z

@pytorchbot merge

(Initiating merge automatically since Phabricator Diff has merged)

pytorchmergebot · 2022-12-21T03:54:57Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

cz-37 requested review from H-Huang, awgu, kwen2501, mrshenli, pritamdamania87, rohan-varma and zhaojuanmao as code owners November 12, 2022 01:11

facebook-github-bot added the fb-exported label Nov 12, 2022

cz-37 force-pushed the export-D41204028 branch from 57015d9 to 31592d2 Compare November 14, 2022 11:01

cz-37 force-pushed the export-D41204028 branch from 31592d2 to eac44f0 Compare November 14, 2022 11:35

d4l3k reviewed Nov 19, 2022

View reviewed changes

cz-37 force-pushed the export-D41204028 branch from eac44f0 to a544a41 Compare November 29, 2022 12:44

cz-37 force-pushed the export-D41204028 branch from a544a41 to f64e446 Compare December 1, 2022 05:02

cz-37 requested a review from wanchaol as a code owner December 1, 2022 05:02

cz-37 force-pushed the export-D41204028 branch from f64e446 to 7ec0fa3 Compare December 5, 2022 21:43

d4l3k approved these changes Dec 16, 2022

View reviewed changes

cz-37 force-pushed the export-D41204028 branch from 7ec0fa3 to 84a0ee6 Compare December 19, 2022 13:38

cz-37 force-pushed the export-D41204028 branch from 84a0ee6 to b2c2025 Compare December 20, 2022 09:50

cz-37 force-pushed the export-D41204028 branch from b2c2025 to 92f310e Compare December 20, 2022 22:52

cz-37 changed the title ~~Update dynamic renderzvous nodes to use rendezvous hostname if provided~~ Update torchrun and TorchElastic to take optional local_addr param to allow skip local IP lookup if specified Dec 20, 2022

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Dec 21, 2022

pytorchmergebot added the Merged label Dec 21, 2022

pytorchmergebot closed this in 5d37890 Dec 21, 2022

janEbert mentioned this pull request Jun 30, 2023

Fix hostname endpoint mismatch #81691

Closed

VeryLazyBoy mentioned this pull request Oct 5, 2023

[BUG] Deepspeed elastic launcher is incompatible with torch 2.0 deepspeedai/DeepSpeed#4459

Closed

xkszltl mentioned this pull request Oct 11, 2023

Getting "master_addr is only used for static rdzv_backend and when rdzv_endpoint is not specified" warning when using rdzv. #111085

Open

Update torchrun and TorchElastic to take optional local_addr param to allow skip local IP lookup if specified #88922

Update torchrun and TorchElastic to take optional local_addr param to allow skip local IP lookup if specified #88922

Uh oh!

Conversation

cz-37 commented Nov 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linux-foundation-easycla bot commented Nov 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/88922

✅ No Failures

Uh oh!

facebook-github-bot commented Nov 12, 2022

Uh oh!

facebook-github-bot commented Nov 14, 2022

Uh oh!

facebook-github-bot commented Nov 14, 2022

Uh oh!

d4l3k Nov 19, 2022

Choose a reason for hiding this comment

Uh oh!

d4l3k Nov 19, 2022

Choose a reason for hiding this comment

Uh oh!

d4l3k Nov 19, 2022

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Nov 29, 2022

Uh oh!

facebook-github-bot commented Dec 1, 2022

Uh oh!

facebook-github-bot commented Dec 5, 2022

Uh oh!

d4l3k left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Dec 19, 2022

Uh oh!

facebook-github-bot commented Dec 20, 2022

Uh oh!

facebook-github-bot commented Dec 20, 2022

Uh oh!

facebook-github-bot commented Dec 21, 2022

Uh oh!

pytorchmergebot commented Dec 21, 2022

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Update torchrun and TorchElastic to take optional `local_addr` param to allow skip local IP lookup if specified #88922

Update torchrun and TorchElastic to take optional `local_addr` param to allow skip local IP lookup if specified #88922

cz-37 commented Nov 12, 2022 •

edited

Loading

linux-foundation-easycla bot commented Nov 12, 2022 •

edited

Loading

pytorch-bot bot commented Nov 12, 2022 •

edited

Loading