[Pytorch] Enable autovec on aarch64 for type conversion#166049
Closed
Nicoshev wants to merge 1 commit intopytorch:mainfrom
Closed
[Pytorch] Enable autovec on aarch64 for type conversion#166049Nicoshev wants to merge 1 commit intopytorch:mainfrom
Nicoshev wants to merge 1 commit intopytorch:mainfrom
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/166049
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 New FailuresAs of commit 4614763 with merge base 13cda9b ( NEW FAILURES - The following jobs have failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
pytorch-bot
bot
added
the
module: cpu
Contributor
Author
|
@pytorchbot label "topic: not user facing" |
Nicoshev
added
ciflow/trunk
Nicoshev
added a commit
to Nicoshev/pytorch
that referenced
this pull request
Oct 22, 2025
Summary: Implementing autovec template for type conversions on aarch64-NEON We've seen significant performance improvements for converting to and from bytes: Before float->uint8->float ===> 683.212us float->int8->float ===> 687.846us int32->uint8->int32 ===> 497.121us int32->int8->int32 ===> 481.889us After: float->uint8->float ===> 198.204us ----> 245% higher throughput float->int8->float ===> 200.241us ----> 244% higher throughput int32->uint8->int32 ===> 197.970us ----> 151% higher throughput int32->int8->int32 ===> 198.206us ----> 143% higher throughput Test Plan: Correctness: buck2 test mode/opt //caffe2/test:test_ops buck2 test mode/opt //caffe2/test:torch Performance: buck2 run mode/opt //caffe2/benchmarks/operator_benchmark/fb:operator_benchmark_test Differential Revision: D85213420
Nicoshev
force-pushed
the
export-D85213420
branch
2 times, most recently
from
34ffa32 to
7b0e870
Compare
Nicoshev
added a commit
to Nicoshev/pytorch
that referenced
this pull request
Oct 22, 2025
Summary: Implementing autovec template for type conversions on aarch64-NEON We've seen significant performance improvements for converting to and from bytes: Before float->uint8->float ===> 683.212us float->int8->float ===> 687.846us int32->uint8->int32 ===> 497.121us int32->int8->int32 ===> 481.889us After: float->uint8->float ===> 198.204us ----> 245% higher throughput float->int8->float ===> 200.241us ----> 244% higher throughput int32->uint8->int32 ===> 197.970us ----> 151% higher throughput int32->int8->int32 ===> 198.206us ----> 143% higher throughput Test Plan: Correctness: buck2 test mode/opt //caffe2/test:test_ops buck2 test mode/opt //caffe2/test:torch Performance: buck2 run mode/opt //caffe2/benchmarks/operator_benchmark/fb:operator_benchmark_test Differential Revision: D85213420
Nicoshev
force-pushed
the
export-D85213420
branch
from
7b0e870 to
cf3fe30
Compare
Nicoshev
added a commit
to Nicoshev/pytorch
that referenced
this pull request
Oct 22, 2025
Summary: Implementing autovec template for type conversions on aarch64-NEON We've seen significant performance improvements for converting to and from bytes: Before float->uint8->float ===> 683.212us float->int8->float ===> 687.846us int32->uint8->int32 ===> 497.121us int32->int8->int32 ===> 481.889us After: float->uint8->float ===> 198.204us ----> 245% higher throughput float->int8->float ===> 200.241us ----> 244% higher throughput int32->uint8->int32 ===> 197.970us ----> 151% higher throughput int32->int8->int32 ===> 198.206us ----> 143% higher throughput Test Plan: Correctness: buck2 test mode/opt //caffe2/test:test_ops buck2 test mode/opt //caffe2/test:torch Performance: buck2 run mode/opt //caffe2/benchmarks/operator_benchmark/fb:operator_benchmark_test Differential Revision: D85213420
Nicoshev
added a commit
to Nicoshev/pytorch
that referenced
this pull request
Oct 22, 2025
Summary: Implementing autovec template for type conversions on aarch64-NEON We've seen significant performance improvements for converting to and from bytes: Before float->uint8->float ===> 683.212us float->int8->float ===> 687.846us int32->uint8->int32 ===> 497.121us int32->int8->int32 ===> 481.889us After: float->uint8->float ===> 198.204us ----> 245% higher throughput float->int8->float ===> 200.241us ----> 244% higher throughput int32->uint8->int32 ===> 197.970us ----> 151% higher throughput int32->int8->int32 ===> 198.206us ----> 143% higher throughput Test Plan: Correctness: buck2 test mode/opt //caffe2/test:test_ops buck2 test mode/opt //caffe2/test:torch Performance: buck2 run mode/opt //caffe2/benchmarks/operator_benchmark/fb:operator_benchmark_test Differential Revision: D85213420
Nicoshev
force-pushed
the
export-D85213420
branch
from
cf3fe30 to
edb1c73
Compare
Nicoshev
force-pushed
the
export-D85213420
branch
from
edb1c73 to
06ef008
Compare
Nicoshev
added a commit
to Nicoshev/pytorch
that referenced
this pull request
Oct 22, 2025
Summary: Implementing autovec template for type conversions on aarch64-NEON We've seen significant performance improvements for converting to and from bytes: Before float->uint8->float ===> 683.212us float->int8->float ===> 687.846us int32->uint8->int32 ===> 497.121us int32->int8->int32 ===> 481.889us After: float->uint8->float ===> 198.204us ----> 245% higher throughput float->int8->float ===> 200.241us ----> 244% higher throughput int32->uint8->int32 ===> 197.970us ----> 151% higher throughput int32->int8->int32 ===> 198.206us ----> 143% higher throughput Test Plan: Correctness: buck2 test mode/opt //caffe2/test:test_ops buck2 test mode/opt //caffe2/test:torch Performance: buck2 run mode/opt //caffe2/benchmarks/operator_benchmark/fb:operator_benchmark_test Differential Revision: D85213420
Nicoshev
force-pushed
the
export-D85213420
branch
from
06ef008 to
cac4953
Compare
Nicoshev
added a commit
to Nicoshev/pytorch
that referenced
this pull request
Oct 22, 2025
Summary: Implementing autovec template for type conversions on aarch64-NEON We've seen significant performance improvements for converting to and from bytes: Before float->uint8->float ===> 683.212us float->int8->float ===> 687.846us int32->uint8->int32 ===> 497.121us int32->int8->int32 ===> 481.889us After: float->uint8->float ===> 198.204us ----> 245% higher throughput float->int8->float ===> 200.241us ----> 244% higher throughput int32->uint8->int32 ===> 197.970us ----> 151% higher throughput int32->int8->int32 ===> 198.206us ----> 143% higher throughput Test Plan: Correctness: buck2 test mode/opt //caffe2/test:test_ops buck2 test mode/opt //caffe2/test:torch Performance: buck2 run mode/opt //caffe2/benchmarks/operator_benchmark/fb:operator_benchmark_test Differential Revision: D85213420
malfet
reviewed
Oct 22, 2025
malfet
added
topic: improvements
Contributor
If it's not user facing, then what's the point of this change? (Labels should be "release notes: cpu (aarch64)" / "topic: performance" |
Nicoshev
force-pushed
the
export-D85213420
branch
from
cac4953 to
0b4be2e
Compare
Nicoshev
added a commit
to Nicoshev/pytorch
that referenced
this pull request
Oct 24, 2025
Summary: Implementing autovec template for type conversions on aarch64-NEON We've seen significant performance improvements for converting to and from bytes: Before float->uint8->float ===> 683.212us float->int8->float ===> 687.846us int32->uint8->int32 ===> 497.121us int32->int8->int32 ===> 481.889us After: float->uint8->float ===> 198.204us ----> 245% higher throughput float->int8->float ===> 200.241us ----> 244% higher throughput int32->uint8->int32 ===> 197.970us ----> 151% higher throughput int32->int8->int32 ===> 198.206us ----> 143% higher throughput Test Plan: Correctness: buck2 test mode/opt //caffe2/test:test_ops buck2 test mode/opt //caffe2/test:torch Performance: buck2 run mode/opt //caffe2/benchmarks/operator_benchmark/fb:operator_benchmark_test Reviewed By: mcfi Differential Revision: D85213420
Nicoshev
added a commit
to Nicoshev/pytorch
that referenced
this pull request
Oct 24, 2025
Summary: Implementing autovec template for type conversions on aarch64-NEON We've seen significant performance improvements for converting to and from bytes: Before float->uint8->float ===> 683.212us float->int8->float ===> 687.846us int32->uint8->int32 ===> 497.121us int32->int8->int32 ===> 481.889us After: float->uint8->float ===> 198.204us ----> 245% higher throughput float->int8->float ===> 200.241us ----> 244% higher throughput int32->uint8->int32 ===> 197.970us ----> 151% higher throughput int32->int8->int32 ===> 198.206us ----> 143% higher throughput Test Plan: Correctness: buck2 test mode/opt //caffe2/test:test_ops buck2 test mode/opt //caffe2/test:torch Performance: buck2 run mode/opt //caffe2/benchmarks/operator_benchmark/fb:operator_benchmark_test Reviewed By: mcfi Differential Revision: D85213420
Nicoshev
force-pushed
the
export-D85213420
branch
from
38d294a to
21031d2
Compare
Summary: Implementing autovec template for type conversions on aarch64-NEON We've seen significant performance improvements for converting to and from bytes: Before float->uint8->float ===> 683.212us float->int8->float ===> 687.846us int32->uint8->int32 ===> 497.121us int32->int8->int32 ===> 481.889us After: float->uint8->float ===> 198.204us ----> 245% higher throughput float->int8->float ===> 200.241us ----> 244% higher throughput int32->uint8->int32 ===> 197.970us ----> 151% higher throughput int32->int8->int32 ===> 198.206us ----> 143% higher throughput Test Plan: Correctness: buck2 test mode/opt //caffe2/test:test_ops buck2 test mode/opt //caffe2/test:torch Performance: buck2 run mode/opt //caffe2/benchmarks/operator_benchmark/fb:operator_benchmark_test Reviewed By: mcfi Differential Revision: D85213420
Nicoshev
force-pushed
the
export-D85213420
branch
from
21031d2 to
9287ff7
Compare
Nicoshev
added a commit
to Nicoshev/pytorch
that referenced
this pull request
Oct 24, 2025
Summary: Implementing autovec template for type conversions on aarch64-NEON We've seen significant performance improvements for converting to and from bytes: Before float->uint8->float ===> 683.212us float->int8->float ===> 687.846us int32->uint8->int32 ===> 497.121us int32->int8->int32 ===> 481.889us After: float->uint8->float ===> 198.204us ----> 245% higher throughput float->int8->float ===> 200.241us ----> 244% higher throughput int32->uint8->int32 ===> 197.970us ----> 151% higher throughput int32->int8->int32 ===> 198.206us ----> 143% higher throughput Test Plan: Correctness: buck2 test mode/opt //caffe2/test:test_ops buck2 test mode/opt //caffe2/test:torch Performance: buck2 run mode/opt //caffe2/benchmarks/operator_benchmark/fb:operator_benchmark_test Reviewed By: mcfi Differential Revision: D85213420
malfet
requested changes
Oct 24, 2025
Contributor
malfet
left a comment
malfet
left a comment
There was a problem hiding this comment.
PR title claims throughput improvements, but does not update any benchmark files.
Moreover, convertImpl feels just like a code duplicate of default template implementation already, isn't it?
Nicoshev
force-pushed
the
export-D85213420
branch
from
9287ff7 to
4614763
Compare
Contributor
Author
@malfet Improvements are observed when targeting SVE, so there was no point in adding a specific benchmark for the OS repo. It is not a duplicate of the existing implementation, as the internally supplied benchmarks show |
Nicoshev
requested review from
malfet and
mcfi
and removed request for
malfet
ezyang
approved these changes
Oct 24, 2025
mcfi
approved these changes
Oct 24, 2025
aditew01
approved these changes
Oct 24, 2025
Contributor
|
@pytorchbot merge (Initiating merge automatically since Phabricator Diff has merged) |
|
This PR has pending changes requested. Please address the comments and update the PR before merging. |
Contributor
Author
|
@pytorchbot merge -f "Benchmark failures are a pre-existing issue" |
Collaborator
Merge startedYour change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Nicoshev
added
the
module: arm
Contributor
|
@pytorchbot revert -c nosignal -m "broke arm builds" |
Collaborator
|
@pytorchbot successfully started a revert job. Check the current status here. |
Collaborator
Reverting PR 166049 failedReason: Command Details for Dev Infra teamRaised by workflow job |
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Implementing autovec template for type conversions on aarch64-NEON
Generated code can be seen here: https://godbolt.org/z/1K6T1d9TE
We've seen significant performance improvements for converting to and from bytes, compiling using clang with -march=armv9-a+sve2:
Before
float->uint8->float ===> 683.212us
float->int8->float ===> 687.846us
int32->uint8->int32 ===> 497.121us
int32->int8->int32 ===> 481.889us
After:
float->uint8->float ===> 198.204us ----> 245% higher throughput
float->int8->float ===> 200.241us ----> 244% higher throughput
int32->uint8->int32 ===> 197.970us ----> 151% higher throughput
int32->int8->int32 ===> 198.206us ----> 143% higher throughput
Test Plan:
buck2 test mode/opt //caffe2/test:test_ops
buck2 test mode/opt //caffe2/test:torch
Differential Revision: D85213420
cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @jerryzh168 @aditew01 @snadampal @milpuz01 @nikhil-arm @fadara01