Skip to content

[dbsp] Avoid one data copy on remote exchange.#5942

Open
blp wants to merge 1 commit intomainfrom
less-copying
Open

[dbsp] Avoid one data copy on remote exchange.#5942
blp wants to merge 1 commit intomainfrom
less-copying

Conversation

@blp
Copy link
Copy Markdown
Member

@blp blp commented Mar 27, 2026

The serializer we use for rkyv outputs into an FBuf, which has 512-byte alignment (because we think we might want to use O_DIRECT someday, which requires 4096- or 512-byte alignment depending on the device). However, our exchange code was then copying the FBuf's contents into a Vec. This commit eliminates the copy by using the FBuf for exchange directly instead of Vec.

(It's questionable whether there's benefit to using FBuf at all. We might be able to get rid of it.)

Describe Manual Test Plan

I ran the dbsp unit tests.

The serializer we use for `rkyv` outputs into an FBuf, which has 512-byte
alignment (because we think we might want to use O_DIRECT someday, which
requires 4096- or 512-byte alignment depending on the device).  However,
our exchange code was then copying the FBuf's contents into a Vec<u8>.
This commit eliminates the copy by using the FBuf for exchange directly
instead of Vec<u8>.

(It's questionable whether there's benefit to using FBuf at all.  We might
be able to get rid of it.)

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp blp requested a review from ryzhyk March 27, 2026 23:12
@blp blp self-assigned this Mar 27, 2026
@blp blp added performance rust Pull requests that update Rust code multihost Related to multihost or distributed pipelines labels Mar 27, 2026
Copy link
Copy Markdown
Contributor

@ryzhyk ryzhyk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Who could say no to eliminating a copy, especially if it's such a simple fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

multihost Related to multihost or distributed pipelines performance rust Pull requests that update Rust code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants