[RFC] Add experimental Pallas TorchInductor backend#166822
[RFC] Add experimental Pallas TorchInductor backend#166822oulgen wants to merge 11 commits intogh/oulgen/7/basefrom
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/166822
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit fac62ad with merge base d980d8d ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
@oulgen , can I expect the Pallas to deliver a more competitive performance advantage compared to Gluon? |
No clue, probably not though considering gluon is a much lower level language able to express hardware semantics better |
torch/_inductor/codegen/pallas.py
Outdated
| - Compute expression with Python operators (compatible with jax.numpy broadcasting) | ||
| - Store as full-array ref assignment: "out_ptrY[...] = <expr>" | ||
| - Generate Python code that defines a Pallas kernel and a host entrypoint. | ||
| - Use async_compile.cutedsl path to compile and load Python code (generic wrapper). |
torch/_inductor/codegen/pallas.py
Outdated
| # Pallas refs must be unpacked with [...] to load the array | ||
| return self.cse.generate( | ||
| self.compute, | ||
| f"{buf}[...]", |
There was a problem hiding this comment.
Add an assert based on index so this errors if the load order is not contiguous.
torch/_inductor/codegen/pallas.py
Outdated
| out = self.args.output(name) | ||
| self.store_buffer_names.add(name) | ||
| # Pallas refs must use [...] assignment to store back to the ref | ||
| self.stores.writeline(f"{out}[...] = {value}") |
There was a problem hiding this comment.
Add an assert based on index so this errors if the load order is not contiguous. Use a shared indexing helper to compute the "..."
| @classmethod | ||
| def get_backend_features(cls, device: torch.device) -> OrderedSet[BackendFeature]: | ||
| # Start minimal: no special features advertised | ||
| return OrderedSet() |
There was a problem hiding this comment.
When you do reductions, consider reduce to single element here if that is something pallas can do fast. Basically, should we break single element output reductions into multiple kernels.
torch/utils/_pallas.py
Outdated
| if not has_pallas_package(): | ||
| return False | ||
|
|
||
| import torch |
There was a problem hiding this comment.
You can import torch in global scope
| cuda_backends = { | ||
| "triton": CUDACombinedScheduling, | ||
| "halide": HalideScheduling, | ||
| "pallas": PallasScheduling, |
There was a problem hiding this comment.
Why is Pallas registered as a cuda backend? Asking from a technical perspective; for example, is this a placeholder, or perhaps the concrete backend/HW diff doesn't ammeter at this layer?
There was a problem hiding this comment.
We can add pallas to other backends too (See halide in both cpu and gpu), i only added to cuda here because i was testing on cuda only for now. Once we have a tpu backend we can test on, we would register pallas to tpu device as well
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Stack from ghstack (oldest at bottom):
Very simple Pallas TorchInductor backend
Given
it outputs
cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben @mlazos