adapters: parallel delta output encoder by swanandx · Pull Request #5869 · feldera/feldera

swanandx · 2026-03-19T16:07:20Z

Parallelize Delta table output encoding by splitting batches across configurable worker threads, each writing independent Parquet files concurrently
Add threads config option (default: 1) to DeltaTableWriterConfig to control the number of parallel encoding threads

Describe Manual Test Plan

Added unit tests and ran locally

for write retry, I changed local permissions for local path to simulate write failure

chmod 000 /tmp/feldera-delta-profiling

and to restore

chmod 755 /tmp/feldera-delta-profiling

& commit retries were tried by renaming the directory

what can be tested further:

on fatal error do we want to stop whole pipeline, just stall it or do something else?
try this out with IRSA and expiring credentials?

Checklist

Unit tests added/updated
Integration tests added/updated
Documentation updated
Changelog updated

Breaking Changes?

we're adding new config options for delta connector but should not be a breaking change.

mythical-fred

Three hard blocks, one correctness concern.

crates/adapters/src/integrated/delta_table/output.rs

crates/feldera-types/src/transport/delta_table.rs

crates/adapters/src/integrated/delta_table/output.rs

mythical-fred

Two code bugs (inline) plus missing docs — blocking on the docs.

threads and fatal_on_max_retries are new user-visible config options but neither appears in docs.feldera.com/docs/connectors/sinks/delta.md. The Documentation checkbox is unchecked. Please add entries to the config table for both options.

crates/feldera-types/src/transport/delta_table.rs

crates/adapters/src/integrated/delta_table/output.rs

Use SplitCursor to split the batch and distribute it across tasks, each task retries encoding and writing to delta lake and then returns Add actions which main task retries to commit to delta lake Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>

ryzhyk

Since we don't do anything meaningful with fatal errors today, let's not bother adding that option. We'll do another pass to rework connector error handling later.

crates/feldera-types/src/transport/delta_table.rs

crates/adapters/src/integrated/delta_table/output.rs

ryzhyk · 2026-03-24T18:07:09Z

crates/adapters/src/integrated/delta_table/output.rs

+    }
+
+    // prints timing for manual comparison, and asserts output correctness
+    #[test]


Let's turn this into a proper benchmark under adapters/bench.
What benchmark results did you get?

$cargo bench --package dbsp_adapters --bench delta_encoder --features with-deltalake

Running benches/delta_encoder.rs (target/release/deps/delta_encoder-836871c7c7c0c9cd) Gnuplot not found, using plotters backend delta_indexed_encode/workers/1 time: [35.170 ms 35.201 ms 35.233 ms] thrpt: [2.8382 Melem/s 2.8408 Melem/s 2.8433 Melem/s] Found 1 outliers among 100 measurements (1.00%) 1 (1.00%) high mild delta_indexed_encode/workers/2 time: [24.756 ms 25.310 ms 25.873 ms] thrpt: [3.8650 Melem/s 3.9511 Melem/s 4.0394 Melem/s] Found 1 outliers among 100 measurements (1.00%) 1 (1.00%) high mild delta_indexed_encode/workers/4 time: [15.615 ms 15.899 ms 16.199 ms] thrpt: [6.1734 Melem/s 6.2897 Melem/s 6.4042 Melem/s] Found 2 outliers among 100 measurements (2.00%) 2 (2.00%) high mild delta_indexed_encode/workers/8 time: [9.1359 ms 9.2460 ms 9.3869 ms] thrpt: [10.653 Melem/s 10.816 Melem/s 10.946 Melem/s] Found 3 outliers among 100 measurements (3.00%) 1 (1.00%) low mild 1 (1.00%) high mild 1 (1.00%) high severe

crates/adapters/src/integrated/delta_table/output.rs

mythical-fred

My previous two blockers (schema minimum = 1 and docs for threads) are resolved — thank you. One remaining issue:

mythical-fred · 2026-03-26T01:11:40Z

crates/feldera-types/src/transport/delta_table.rs

    pub max_retries: Option<u32>,

+    /// Number of parallel threads used by the connector.
+    ///


Commit message requested changes on the latest commit is a hard block. Please squash or rebase: git rebase -i and rewrite it with an imperative summary describing what changed, e.g. [adapters] Use Option<usize> for threads; remove fatal_on_max_retries. Resources: https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html

ryzhyk

Thanks for addressing all comments! I only have a couple of questions left (inline)

ryzhyk · 2026-03-27T15:03:49Z

crates/adapters/src/integrated/delta_table/output.rs

+        if threads > 1 && key_schema.is_none() {
+            return Err(ControllerError::invalid_transport_configuration(
+                endpoint_name,
+                "parallel writes (threads > 1) require an index (key_schema) to ensure \


"Parallel writes (threads > 1) require the view to have a unique key to ensure correct ordering of inserts and deletes. Please specify the `index` property in the connector configuration. For more details, see: https://docs.feldera.com/connectors/unique_keys"

ryzhyk · 2026-03-27T15:11:24Z

crates/adapters/src/integrated/delta_table/output.rs

+    //   cargo bench -p dbsp_adapters --bench delta_encoder
    #[test]
    #[ignore]
    fn bench_parallel_encoding() {


why keep the test and why #[ignore] it? If it improves test coverage, perhaps we can make a version with less data that can run as a test?

ryzhyk · 2026-03-27T15:12:58Z

crates/adapters/src/integrated/delta_table/test.rs

+    let mut storage_options: serde_json::Value = serde_json::to_value(object_store_config).unwrap();
+    storage_options["uri"] = json!(table_uri);
+    storage_options["mode"] = json!("truncate");
+    if let Some(threads) = threads {


I don't see it used in any test.

swanandx force-pushed the parallel-delta-output branch 3 times, most recently from 56f4f84 to 52065ab Compare March 23, 2026 13:06

swanandx requested a review from mythical-fred March 23, 2026 13:10

swanandx marked this pull request as ready for review March 23, 2026 13:10

mythical-fred suggested changes Mar 23, 2026

View reviewed changes

crates/feldera-types/src/transport/delta_table.rs Outdated Show resolved Hide resolved

crates/adapters/src/integrated/delta_table/output.rs Outdated Show resolved Hide resolved

ryzhyk self-requested a review March 23, 2026 19:04

swanandx force-pushed the parallel-delta-output branch from 6738817 to ac1dfbd Compare March 24, 2026 06:31

swanandx requested a review from mythical-fred March 24, 2026 06:32

swanandx force-pushed the parallel-delta-output branch from ac1dfbd to adf3e2c Compare March 24, 2026 06:52

ryzhyk requested changes Mar 24, 2026

View reviewed changes

requested changes

47b7efc

swanandx requested review from mythical-fred and ryzhyk and removed request for mythical-fred March 25, 2026 16:47

mythical-fred suggested changes Mar 26, 2026

View reviewed changes

ryzhyk reviewed Mar 27, 2026

View reviewed changes

Conversation

swanandx commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe Manual Test Plan

Checklist

Breaking Changes?

Uh oh!

mythical-fred left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mythical-fred left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ryzhyk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryzhyk Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

swanandx Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mythical-fred left a comment

Choose a reason for hiding this comment

Uh oh!

mythical-fred Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

ryzhyk left a comment

Choose a reason for hiding this comment

Uh oh!

ryzhyk Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

ryzhyk Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

ryzhyk Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

swanandx commented Mar 19, 2026 •

edited

Loading