Thread deterministic config vars to subproc compilation#165729
Thread deterministic config vars to subproc compilation#165729drisspg wants to merge 5 commits intogh/drisspg/210/basefrom
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/165729
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 477046a with merge base 36371b8 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
drisspg
changed the title
|
config overriding is usually added here: https://github.com/pytorch/pytorch/blob/main/torch/_inductor/codegen/triton.py#L4764 There are already multiple settings there. |
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Pull Request resolved: #165866 Approved by: https://github.com/BoyuanFeng ghstack dependencies: #165729
| @@ -2973,7 +2973,7 @@ def filter_reduction_configs_for_determinism( | |||
| def _do_filter_due_to_inductor_config(): | |||
| return ( | |||
There was a problem hiding this comment.
Can we do anything defensively to avoid this happening again? E.g., factor out torch._inductor.config accesses in this file to a file that asserts that the attribute is in the fields that we patch to subprocess ?
There was a problem hiding this comment.
cc @shunting314
I like this idea and agree we should do something
There was a problem hiding this comment.
I think it's not hard to go thru the file and make sure there is no directly access to torch._inductor.config (e.g. access them thru inductor_meta dict), but it would be hard to
- avoid future code in this file from accessing inductor config directly
- avoid the code this file rely on from accessing inductor config directly. There accesses are harder to find.
Alternatively, maybe we should just detect and patch every config overriding when submitting a compilation task to child process.
# Summary TIL (AFTER WAYYYY TOO MUCH INSANITY), that we do not serialize the full set of configs for the subproc compilation. I found this while working on Flex-attention determinism: meta-pytorch/attention-gym#168 might be good to audit if we need to thread through any more Pull Request resolved: pytorch#165729 Approved by: https://github.com/shunting314, https://github.com/eellison
…5866) Pull Request resolved: pytorch#165866 Approved by: https://github.com/BoyuanFeng ghstack dependencies: pytorch#165729
# Summary TIL (AFTER WAYYYY TOO MUCH INSANITY), that we do not serialize the full set of configs for the subproc compilation. I found this while working on Flex-attention determinism: meta-pytorch/attention-gym#168 might be good to audit if we need to thread through any more Pull Request resolved: pytorch#165729 Approved by: https://github.com/shunting314, https://github.com/eellison
…5866) Pull Request resolved: pytorch#165866 Approved by: https://github.com/BoyuanFeng ghstack dependencies: pytorch#165729
5 participants
Stack from ghstack (oldest at bottom):
Summary
TIL (AFTER WAYYYY TOO MUCH INSANITY), that we do not serialize the full set of configs for the subproc compilation.
I found this while working on Flex-attention determinism: meta-pytorch/attention-gym#168
might be good to audit if we need to thread through any more
cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben @mlazos