Skip to content

Fragment large TsBlocks of query result responses between java session and server - #19263

Open
Wei-hao-Li wants to merge 9 commits into
masterfrom
fragmentCSRPC
Open

Wei-hao-Li wants to merge 9 commits into
masterfrom
fragmentCSRPC

Conversation

@Wei-hao-Li

Copy link
Copy Markdown
Collaborator

No description provided.

Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
@codecov

codecov Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 68.19222% with 139 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.24%. Comparing base (720da6c) to head (3a83908).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
.../db/protocol/thrift/impl/ClientRPCServiceImpl.java 13.54% 83 Missing ⚠️
...ne/plan/execution/MaterializedResultExecution.java 59.75% 33 Missing ⚠️
...ain/java/org/apache/iotdb/rpc/IoTDBRpcDataSet.java 91.52% 5 Missing ⚠️
...ache/iotdb/db/protocol/session/IClientSession.java 25.00% 3 Missing ⚠️
...java/org/apache/iotdb/jdbc/IoTDBJDBCResultSet.java 60.00% 2 Missing ⚠️
...va/org/apache/iotdb/session/SessionConnection.java 84.61% 2 Missing ⚠️
...yengine/execution/exchange/SharedTsBlockQueue.java 92.00% 2 Missing ⚠️
.../apache/iotdb/db/queryengine/plan/Coordinator.java 0.00% 2 Missing ⚠️
.../db/queryengine/plan/execution/QueryExecution.java 66.66% 2 Missing ⚠️
...java/org/apache/iotdb/isession/SessionDataSet.java 83.33% 1 Missing ⚠️
... and 4 more
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #19263      +/-   ##
============================================
+ Coverage     49.22%   49.24%   +0.02%     
- Complexity      717      718       +1     
============================================
  Files          5646     5648       +2     
  Lines        416368   416693     +325     
  Branches      54966    55011      +45     
============================================
+ Hits         204959   205216     +257     
- Misses       211409   211477      +68     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address the memory accounting and read-size quota regressions introduced by fragmentation, and apply the payload limit to the Fast LAST cache-hit paths. The inline comments describe the affected paths.

…d quota by whole blocks

- SharedTsBlockQueue.remove(retainMemory) keeps the reservation of a block that
  outlives the RPC in PendingResultFragment; freeRetainedMemory() releases it and
  close/abort free the buffered and handed-over memory together.
- LocalSourceHandle delegates the retain/release to its queue, so single-node
  deployments keep fragmented transfers accounted in the memory pool.
- OperationQuota only exposes addReadResult(long bytes); responses charge a block
  whose fragmentation starts in them by its totalLength, so a fragmented result
  is charged exactly once against the read quota.
…result

The cache-hit branches of executeFastLastDataQueryForOneDeviceV2 and
executeFastLastDataQueryForOnePrefixPath serialized the whole result into one
response, so an oversized cached value could still exceed the Thrift frame
limit. The cached result is now registered as a MaterializedResultExecution and
sent through the regular fragmented transfer: fetchResultsV2 continues from its
pending fragment, and the serialized bytes are reserved from the query memory
pool (best effort) until the last fragment is sent or the query is cleaned up.
Conflicts in i18n message classes are additive; kept the constants from both
sides (fragmented query result messages and the incoming buffer/frame-size
messages).
respondWithFragmentedResult is only entered when the cached result exceeds the
payload bound, so the materialized transfer always starts with a fragment;
replace the defensive finish branch with checkState.
Conflicts in the JDBC result-set construction: keep the upstream resource
cleanup (close the query operation when result-set creation fails) together
with the offset/totalLength arguments added for fragmented results.
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants