Skip to content

Issue 68576 prefetch factor #88972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
blankstatic wants to merge 4 commits into from
Closed

Issue 68576 prefetch factor #88972

blankstatic wants to merge 4 commits into from

Conversation

@blankstatic
Copy link
Contributor

@blankstatic blankstatic commented Nov 13, 2022

Fixes #68576
This PR allows set the prefetch_factor=None making it really optional according to the documentation

@pytorch-bot pytorch-bot bot added the release notes: dataloader release notes category label Nov 13, 2022
@pytorch-bot
Copy link

pytorch-bot bot commented Nov 13, 2022 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/88972

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit d6b83a7:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Nov 13, 2022 •
edited
Loading

CLA Signed

The committers listed above are authorized under a signed CLA.

@kit1980
Copy link
Contributor

kit1980 commented Nov 14, 2022

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Nov 14, 2022
@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Nov 14, 2022

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Nov 14, 2022

Merge failed

Reason: The following mandatory check(s) failed (Rule superuser):

Dig deeper by viewing the failures on hud

Details for Dev Infra team Raised by workflow job

@blankstatic blankstatic requested a review from kit1980 November 15, 2022 00:08
@kit1980
Copy link
Contributor

kit1980 commented Nov 16, 2022

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Nov 16, 2022

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Nov 16, 2022

Merge failed

Reason: 2 additional jobs have failed, first few of them are: trunk ,trunk / cuda11.6-py3.10-gcc7-sm86 / test (default, 2, 4, linux.g5.4xlarge.nvidia.gpu)

Details for Dev Infra team Raised by workflow job

@kit1980
Copy link
Contributor

kit1980 commented Nov 17, 2022

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Nov 17, 2022

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Nov 17, 2022

Merge failed

Reason: 2 additional jobs have failed, first few of them are: trunk ,trunk / cuda11.6-py3.10-gcc7-sm86 / test (default, 2, 4, linux.g5.4xlarge.nvidia.gpu)

Details for Dev Infra team Raised by workflow job

@kit1980
Copy link
Contributor

kit1980 commented Nov 17, 2022

@pytorchbot rebase

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Nov 17, 2022

@pytorchbot successfully started a rebase job. Check the current status here

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Nov 17, 2022

Successfully rebased issue-68576-prefetch_factor onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout issue-68576-prefetch_factor && git pull --rebase)

@kit1980
Copy link
Contributor

kit1980 commented Nov 17, 2022

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Nov 17, 2022

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Nov 17, 2022

Merge failed

Reason: The following mandatory check(s) failed (Rule superuser):

Dig deeper by viewing the failures on hud

Details for Dev Infra team Raised by workflow job

@kit1980
Copy link
Contributor

kit1980 commented Nov 17, 2022

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Nov 17, 2022

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

ejguan
ejguan reviewed Nov 18, 2022
View reviewed changes
Copy link
Contributor

@ejguan ejguan left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for adding this PR. Could you please add a follow up PR to Fix the docstring related to prefetch_factor -> Default is None when num_workers=0 but default is 2 when num_workers>0

@ejguan ejguan added the topic: improvements topic category label Nov 18, 2022
@ejguan ejguan mentioned this pull request Nov 18, 2022
Closed
facebook-github-bot pushed a commit to meta-pytorch/data that referenced this pull request Nov 19, 2022
@ejguan @facebook-github-bot
Fix tests for Azure and DataLoader (#905)
67afe0c 
Summary:
### Changes

- For Azure test: Please see #904
- For DataLoader test, pytorch/pytorch#88972 introduces an Error when `prefetch_factor` is specified when `num_workers>0`.

Pull Request resolved: #905

Reviewed By: wenleix

Differential Revision: D41403018

Pulled By: ejguan

fbshipit-source-id: 7f1d0677a7311b5c7b3d72b86501301d7a68e7a7
kulinseth pushed a commit to kulinseth/pytorch that referenced this pull request Dec 10, 2022
@blankstatic @kulinseth
Issue 68576 prefetch factor (pytorch#88972)
5f38155 
Fixes pytorch#68576
This PR allows set the `prefetch_factor=None` making it really optional according to the documentation
Pull Request resolved: pytorch#88972
Approved by: https://github.com/kit1980
@nguyenkhoihoang nguyenkhoihoang mentioned this pull request Jan 31, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ejguan ejguan ejguan left review comments

@kit1980 kit1980 Awaiting requested review from kit1980

Assignees

No one assigned

Labels

ciflow/trunk Trigger trunk jobs on your pull request Merged open source release notes: dataloader release notes category topic: improvements topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

When num_workers=0, prefetch_factor is enforced to be 2 but this is missing from documentation

5 participants

@blankstatic @kit1980 @pytorchmergebot @ejguan @pytorchbot