Skip to content

Optimize asynchronous merger #2115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
blp merged 16 commits into from
Jul 30, 2024
Merged

Optimize asynchronous merger #2115

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
669e1e9
dbsp: Fix comment on `Runtime::run`.
blp Jul 27, 2024
6ba13b6
dbsp: Avoid lots of copying when persisting a spine.
blp Jul 19, 2024
e090bde
benchmark: Make `run-nexmark.sh` accept `--storage` option.
blp Jul 24, 2024
2013e1e
dbsp: Remove "time" parameter from `empty()` and `dyn_empty()` methods.
blp Jul 26, 2024
ef3a8a1
dbsp: Allow `merge_batches` to merge any kind of batch.
blp Jul 26, 2024
9a50824
dbsp: Drop unneeded use of `crossbeam::sync::Parker`.
blp Jul 26, 2024
7574604
dbsp: Use only a single kill signal across all runtime threads.
blp Jul 27, 2024
29407db
dbsp: Remove `id` member from `BufferCache`.
blp Jul 27, 2024
4103f0a
dbsp: Remove unused method `Runtime::background_index()`.
blp Jul 28, 2024
0fab4ff
dbsp: Refactor how async spine connects to its background merger.
blp Jul 28, 2024
682b732
dbsp: Rewrite asynchronous merger to be more asynchronous.
blp Jul 26, 2024
5453f49
dbsp: Avoid cases where empty batches could enter the spine.
blp Jul 29, 2024
795b578
dbsp: Supply capacity for `Vec` allocations in async spine.
blp Jul 29, 2024
92f6029
dbsp: Wait for background threads to exit in `RuntimeHandle::join()`.
blp Jul 29, 2024
f59f4a8
dbsp: Fix waterline map tests.
blp Jul 30, 2024
54256a5
dbsp: Fix test_truncate_value_bounded_memory test with new spine.
blp Jul 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 5 additions & 0 deletions benchmark/run-nexmark.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ if ! cores=$(nproc 2>/dev/null) || test $cores -gt 16; then
cores=16
fi
run=:
storage=false
parse=false

# Feldera SQL options.
Expand Down Expand Up @@ -114,6 +115,9 @@ do
--partitions)
nextarg=partitions
;;
--storage)
storage=:
;;
--project=*)
project=${arg#--project=}
;;
Expand Down Expand Up @@ -508,6 +512,7 @@ case $runner:$language in
--poller-threads 10 \
--input-topic-suffix="-$partitions-$events" \
--csv results.csv \
$(if $storage; then printf "%s" --storage; fi) \
--query $(if test $query = all; then echo all; else echo q$query; fi)
;;

Expand Down
12 changes: 6 additions & 6 deletions crates/dbsp/src/circuit/circuit_builder.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3003,11 +3003,11 @@ where
let exchange_id = runtime.sequence_next(worker_index);
let exchange = Exchange::with_runtime(&runtime, exchange_id);

let unparker = Runtime::parker().with(|parker| parker.unparker().clone());
exchange.register_sender_callback(worker_index, move || unparker.unpark());
let thread = std::thread::current();
exchange.register_sender_callback(worker_index, move || thread.unpark());

let unparker = Runtime::parker().with(|parker| parker.unparker().clone());
exchange.register_receiver_callback(worker_index, move || unparker.unpark());
let thread = std::thread::current();
exchange.register_receiver_callback(worker_index, move || thread.unpark());

let termination_check = move || {
// Send local fixed point status to all peers.
Expand All @@ -3016,7 +3016,7 @@ where
if Runtime::kill_in_progress() {
return Err(SchedulerError::Killed);
}
Runtime::parker().with(|parker| parker.park());
std::thread::park();
}
// Receive the fixed point status of each peer, compute global fixedpoint
// state as a logical and of all peer states.
Expand All @@ -3027,7 +3027,7 @@ where
return Err(SchedulerError::Killed);
}
// Sleep if other threads are still working.
Runtime::parker().with(|parker| parker.park());
std::thread::park();
}
Ok(global_fixedpoint)
};
Expand Down
4 changes: 1 addition & 3 deletions crates/dbsp/src/circuit/dbsp_handle.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -416,9 +416,7 @@ impl Runtime {
}
// Nothing to do: do some housekeeping and relinquish the CPU if there's none
// left.
Err(TryRecvError::Empty) => {
Runtime::parker().with(|parker| parker.park());
}
Err(TryRecvError::Empty) => std::thread::park(),
Err(_) => {
break;
}
Expand Down
Loading