perf: add C-contiguous fast path to toBuffer() eliminating per-call allocation - #35
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a C-contiguous fast path to
toBuffer()(andintoBuffer()) that copies data directly from the underlying Rust buffer via memcpy, skipping the intermediate full-array allocation and copy thatextract_array_*incurred on every call.Motivation and Context
fromBuffer()has always been fast: one bulk memcpy into internal storage.toBuffer()was significantly slower and degraded as array sizes grew, making it a bottleneck in ML inference pipelines (e.g. ONNX runtime output buffers, growing KV caches).Root cause: every
toBuffer()call ranextract_array_*which allocated a new Vec and copied the entire array — even when the data was already C-contiguous and could be read in-place. The subsequentcopy_array_to_bufferalso walked elements one at a time via stride iteration instead of using a bulk slice copy.What's Changed
fromBuffer())extract_array_*pathcopy_array_to_buffernow usesas_slice()+copy_from_sliceinstead of element-by-element stride iterationBreaking Changes
None