peer: Sync blocks from multiple peers in parallel - #31
Open
pzafonte wants to merge 1 commit into
Open
Conversation
Contributor
|
Converting to draft because this depends on 65bf637 |
rustaceanrob
marked this pull request as draft
March 5, 2026 14:25
pzafonte
force-pushed
the
multi-peer-ibd-v3
branch
from
March 26, 2026 14:51
5e51121 to
ca0c06b
Compare
pzafonte
changed the title
pzafonte
marked this pull request as ready for review
March 26, 2026 15:43
pzafonte
marked this pull request as draft
July 1, 2026 16:24
pzafonte
force-pushed
the
multi-peer-ibd-v3
branch
from
August 4, 2026 20:01
ca0c06b to
d85b6f7
Compare
pzafonte
changed the title
pzafonte
force-pushed
the
multi-peer-ibd-v3
branch
from
August 5, 2026 13:48
d85b6f7 to
d646771
Compare
Syncing from one peer is only as fast as that peer. Connect to several at once and share the work of downloading blocks between them, so a slow or unresponsive peer no longer holds up the whole sync. A shared queue and an in-flight set coordinate the peers: each block is fetched by exactly one peer, and any block left behind when a peer drops returns to the queue for another to fetch, so none is stranded. Peers fetch blocks out of order, but the kernel must connect them in chain order. A shared buffer holds each arriving block until its parent has been handed off, then releases the unbroken run that now extends the tip. Without it the sync stalls the moment one peer holds a block whose parent another peer has not yet delivered.
pzafonte
force-pushed
the
multi-peer-ibd-v3
branch
from
August 12, 2026 14:38
d646771 to
243df2d
Compare
pzafonte
marked this pull request as ready for review
August 12, 2026 17:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the foundation for multi-peer IBD in kernel-node. Refinements can, will, and should follow as separate changes.
Initial block download fetches blocks from a single peer, so the sync runs at that peer's pace and stalls when it is slow or unresponsive.
Connect to several peers at once, four by default, and split the block download between them. A shared queue hands out batches of hashes and an in-flight set keeps two peers from requesting the same block. When a peer disconnects, the blocks it still owed return to the queue.
Peers reply out of order, but the kernel connects blocks in chain order. Arriving blocks are buffered by parent hash and released as an unbroken run once their parent has been handed off. Without this the sync stops the moment one peer holds a block whose parent another peer has not yet delivered.
This PR is following Bitcoin Core's behavior in the following ways:
DOWNLOAD_BATCH_SIZE = 16MAX_BLOCKS_IN_TRANSIT_PER_PEERDownloadState::in_flightmapBlocksInFlightrequeue_unreceivedFinalizeNodeIt diverges from Core when it comes to block ordering. Core writes blocks as they arrive and lets chain activation order them, bounded by
BLOCK_DOWNLOAD_WINDOW. The kernel needs blocks handed over in chain order, so I am doingbuffer_and_drain.Also of note, recently, shutdown and stale-block detection previously closed the single peer's connection to unblock its read (#90). With several connections there is no one connection to close, so the peer manager signals every thread and exposes the per-peer writers that both paths now use.
Possibly Useful Additions to The Kernel:
btck_ValidationInterfaceActiveTipChange- Tells us when the tip moves allowing the node to follow the kernel instead of keeping a copy.btck_chainstate_manager_is_initial_block_download- Maybe this would be better than usingSynchronizationState, which saves the value from a notification, to tell us whether the node is still catching up?btck_ValidationInterfaceUpdatedBlockTip- It carries the fork point, but not sure if it's useful beyond that.