Skip to content

Add staged test insertion APIs for Java tree and table sessions - #19308

Open
hongzhi-gao wants to merge 4 commits into
apache:masterfrom
hongzhi-gao:feat/staged-test-insert-latest
Open

hongzhi-gao wants to merge 4 commits into
apache:masterfrom
hongzhi-gao:feat/staged-test-insert-latest

Conversation

@hongzhi-gao

@hongzhi-gao hongzhi-gao commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Description

Existing tree-model test insertion APIs only acknowledge reception by the server. This PR adds explicit completion stages so callers can exercise decoding, read-only analysis, distributed planning, and delivery to write targets without inserting data or changing metadata.

Public Java APIs

Add eight stage-aware overloads to ISession / Session:

  • testInsertTablet(Tablet, TestInsertStage) and testInsertTablet(Tablet, boolean sorted, TestInsertStage).
  • testInsertTablets(Map<String, Tablet>, TestInsertStage) and the overload with boolean sorted.
  • testInsertRecord with a trailing TestInsertStage, for both string values and explicit data types/values.
  • testInsertRecords with a trailing TestInsertStage, for both batched string values and explicit data types/values.

Add ITableSession.testInsert(Tablet) and ITableSession.testInsert(Tablet, TestInsertStage), implemented by TableSession and pooled table sessions. The default table overload stops at UNTIL_RPC_PROCESSED. Session.testInsertRelationalTablet(Tablet, TestInsertStage) provides the underlying table-model implementation. Existing tree-model overloads retain their behavior.

Completion boundaries

Stage Acknowledged work
UNTIL_RPC_PROCESSED Login, stage and payload-count checks; no Statement decoding
UNTIL_DECODED Insertion payload decoded into a Statement
UNTIL_ANALYZED Read-only analysis completed
UNTIL_PLANNED Logical and distributed plans generated
UNTIL_WRITE_TARGET_RECEIVED All plan targets have deserialized their plans and checked the insertion plan type and target DataRegion existence

For example, given an existing table/device, unchanged attributes and a data partition covering the tablet timestamps:

// Only acknowledge reception.
tableSession.testInsert(tablet);

// Complete distributed planning without inserting any rows.
tableSession.testInsert(tablet, TestInsertStage.UNTIL_PLANNED);

// Deliver and check all target plans without executing writes.
tableSession.testInsert(tablet, TestInsertStage.UNTIL_WRITE_TARGET_RECEIVED);

Queries return the same rows after each call. If analysis would require a new device, an attribute update, schema creation, or a new data partition, analysis and later stages fail instead of performing that change.

Execution and compatibility

  • Add client RPC testInsert(TSTestInsertReq), response TSTestInsertResp, and TestInsertStage.
  • Add internal RPC testInsertPlanNodes(TSendBatchPlanNodeReq) for target validation. Separate RPCs prevent older nodes from silently ignoring a test flag and executing a write.
  • Check the acknowledged stage on the client. An unsupported server fails explicitly; there is no fallback to normal insertion.
  • Reuse the tree/table planners while guarding metadata mutation paths and using read-only partition lookup.
  • Reject filtered-empty staged tablet requests locally with StatementExecutionException, matching the staged record APIs, before any RPC. Preserve the column categories generated for the filtered table-model payload.
  • Never enter trigger, consensus, WAL or storage write execution. Release frontend memory reservations and schema read locks in finally.
  • Preserve upstream pool synchronization and close handling; connection failures discard the session, while statement failures allow reuse.
  • Localize new diagnostics in English and Chinese and clarify the documented stage prerequisites.

Targets are checked sequentially. This API does not establish WAL durability, replication success, or production write-throughput performance.

Validation

  • 158 client unit tests passed, including the filtered-empty and column-category regressions; both regressions also passed with with-zh-locale.
  • All four IoTDBStagedTestInsertIT cases passed on 1C1D and 1C3D before the final client fix. That fix was verified with unit tests and live-server checks; the JUnit IT suite was not rerun.
  • English/Chinese reactor test compilation and formatting/static checks passed before the final client fix; the updated client passed compilation and formatting checks.

Table Session request latency

Measured at eb0c06d69f on 2026-11-18: a separate WSL client called a remote 1C3D deployment (all nodes on one server, three replicas). One thread reused a connection; each Tablet contained one device, one TAG and three FIELDs. Each stage/batch size had 500 warmups and 3 × 1,000 measured calls.

Times cover tableSession.testInsert(tablet, stage), including client encoding, network and server processing; connection setup and Tablet construction are excluded. All 30,000 measured requests succeeded, with stored data unchanged.

Stage 1 row: mean / P95 (ms) 1,000 rows: mean / P95 (ms)
UNTIL_RPC_PROCESSED 2.549 / 5.467 4.288 / 7.818
UNTIL_DECODED 2.605 / 5.551 4.365 / 7.810
UNTIL_ANALYZED 2.826 / 6.216 4.451 / 8.170
UNTIL_PLANNED 2.833 / 6.406 4.460 / 8.403
UNTIL_WRITE_TARGET_RECEIVED 3.314 / 7.501 4.808 / 8.431

A separate 100-packet ICMP measurement on the same client/server path gave RTT mean 3.045 ms, P50 2.120 ms, P95 8.170 ms, with no packet loss. It was collected after the API test and is not subtracted from the API timings. These results describe this environment, not isolated phase costs or normal-write throughput.

Review

Independent source review covered metadata mutation paths, partition lookup, target validation, memory/lock cleanup and pool synchronization. A locale-dependent assertion was corrected to use the localized message constant.

This PR has:

  • been self-reviewed and independently reviewed.
  • documented the new APIs and stage boundaries.
  • added unit tests and integration tests.
  • been tested in a test IoTDB cluster.

Key changed classes

ISession, ITableSession, Session, SessionConnection, TableSession, TableSessionWrapper, ClientRPCServiceImpl, Coordinator, MPPQueryContext, ClusterPartitionFetcher, schema validators and TestInsertPlanValidator.

@Caideyipi Caideyipi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of head 7c30137 found one actionable issue; no changes were made. The visible CI checks for this head completed successfully. No local build or tests were run. Requesting changes.

throws IoTDBConnectionException, StatementExecutionException {
Objects.requireNonNull(stage, SessionMessages.EXCEPTION_STAGE_9A8C2B9D);
TSInsertTabletReq request = genTSInsertTabletReq(tablet, false, false);
request.setWriteToTable(true);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

genTSInsertTabletReq returns null when SessionUtils.filterNullColumns removes every FIELD column. This path dereferences that null request, so a staged table-model tablet containing only all-null FIELD columns throws a client-side NullPointerException before the RPC. Please handle the filtered-empty case explicitly and cover it in a test; the tree staged tablet overloads also lack explicit handling for a filtered-empty request.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants