Repository navigation
Prefer the local DataNode when LOAD creates DataRegions - #19291
Open
luoluoyuyu wants to merge 1 commit into
Open
luoluoyuyu wants to merge 1 commit into
luoluoyuyu wants to merge 1 commit into
Conversation
When a LOAD has to create the DataRegions of the TsFile it is loading,
the Region allocator picks the replicas purely by the configured policy
(GCR random weight, GREEDY free disk, PGP subgraph). Data that is
partitioned by DataNode and loaded on the DataNode that produced it
therefore usually ends up on another node, and every read of it has to
cross the network.
LOAD now accepts an opt-in attribute that pins the local DataNode as the
first replica of each DataRegion created by that request:
LOAD '/data/example.tsfile' WITH ('prefer-local-node'='true')
The first replica is the DataNode that the write path routes the
RegionGroup to, so the loaded data is also written locally. The remaining
rf - 1 replicas are still chosen by the configured policy (GCR, GREEDY or
PGP), which keeps all of its balancing keys in charge and only steers one
slot.
The preference is off by default and best-effort:
- it only affects the DataRegions created by that request; existing
Regions are never moved and SchemaRegions are never hinted;
- it is honoured only while the local DataNode is Running and still a
placement candidate, so it can neither place a replica on a DataNode
that went down nor fail an otherwise servable request. A DataNode that
fails while the RegionGroups are being created is repaired by the Region
creation procedure with a regular replica re-allocation;
- the replica order is re-sorted when a replica is added or removed later,
so the locality is best-effort rather than sticky;
- the attribute is carried through statement splitting, the tree-to-table
conversion, the analysis-time tablet conversion, the fallback and
rewrite conversions, the asynchronous load handoff and the active-load
retry path. Pipe-generated loads never set it.
The hint travels as TDataPartitionReq.preferredDataNodeId to the
ConfigNode, where each allocator seeds the pinned replica and keeps its
own policy for the rest of the replica set.
luoluoyuyu
force-pushed
the
feature/load-local-node-affinity
branch
from
November 17, 2026 01:35
b5c8936 to
af2a847
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a LOAD has to create the DataRegions of the TsFile it is loading, the Region allocator picks the replicas purely by the configured policy (GCR random weight, GREEDY free disk, PGP subgraph). Data that is partitioned by DataNode and loaded on the DataNode that produced it therefore usually ends up on another node, and every read of it has to cross the network.
LOAD now accepts an opt-in attribute that pins the local DataNode as the first replica of each DataRegion created by that request:
LOAD '/data/example.tsfile' WITH ('prefer-local-node'='true')
The first replica is the DataNode that the write path routes the RegionGroup to, so the loaded data is also written locally. The remaining rf - 1 replicas are still chosen by the configured policy (GCR, GREEDY or PGP), which keeps all of its balancing keys in charge and only steers one slot.
The preference is off by default and best-effort:
The hint travels as TDataPartitionReq.preferredDataNodeId to the ConfigNode, where each allocator seeds the pinned replica and keeps its own policy for the rest of the replica set.
This PR has:
for an unfamiliar reader.
for code coverage.
Key changed/added classes (or packages if there are too many classes) in this PR