Skip to content

Fix table session pool lifecycle leaks - #19247

Open
shuwenwei wants to merge 14 commits into
masterfrom
fix/table-session-pool-lifecycle
Open

shuwenwei wants to merge 14 commits into
masterfrom
fix/table-session-pool-lifecycle

Conversation

@shuwenwei

@shuwenwei shuwenwei commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Problem

TableSessionPool.getSession() borrowed a raw Session from the underlying SessionPool, but did not register it in occupied. Consequently, pool.close() could not see active table sessions, and returning one after the pool closed could publish the connection back into the closed queue.

Example:

ITableSessionPool pool = new TableSessionPoolBuilder()
    .nodeUrls(nodeUrls)
    .user(user)
    .password(password)
    .maxSize(1)
    .build();

ITableSession session = pool.getSession();
pool.close();
session.close();

Before this change:

  • pool.close() did not close the borrowed table session.
  • session.close() returned it to an already-closed pool.
  • Closing the wrapper after a connection failure could throw an NPE.
  • A failed SQL dialect/database reset could permanently consume pool capacity.
  • pool.close() could hold the pool monitor while waiting for an in-flight RPC, blocking unrelated borrow and release operations.

Tree-model result queries had a related acquisition window. They called occupy(session) only after the query RPC completed. If pool.close() ran in between, the Session could be registered in a pool that had already completed cleanup.

Corrected behavior

  • Borrowed table sessions are tracked in occupied until their wrapper is closed.
  • Releasing into a closed pool closes the Session directly instead of putting it back into the queue.
  • pool.close() snapshots and clears pool state under the monitor, then closes connections outside the monitor.
  • Borrowed sessions are force-closed through their transport so pool shutdown does not wait behind an in-flight statement RPC.
  • Tree-model query completion checks closed before adding the Session to occupied; a concurrent pool close causes the Session to be closed instead.
  • Failed dialect/database resets and connection errors release the unusable Session and restore pool capacity with a replacement when possible.
  • Table-session operations remain serialized, and old tree-model result-set wrappers cannot release a Session that has since been re-borrowed.
  • sortTablet() performs its in-memory sort without borrowing a pooled Session.
  • Reusing a table-session wrapper after a connection failure now raises IoTDBConnectionException instead of an NPE.

Changes

  • Add table-session ownership tracking through occupied.
  • Introduce a shared release path for successful reuse, failed sessions, ResultSets, and table-session wrappers.
  • Add lock-free-of-RPC pool shutdown and Session transport force close.
  • Reject tree-model occupied registration after pool close.
  • Preserve protected API compatibility for SessionPool subclasses.
  • Reconcile the implementation with the pooled-session cleanup and ResultSet detachment fixes already merged into master.
  • Add deterministic latch-based tests for borrow/close and release/close interleavings.
  • Add capacity tests for dialect/database reset and connection failure recovery.
  • Add tests for wrapper reuse after failure, pool close without monitor blocking, tree-query close races, and sortTablet() behavior.

Validation

  • git diff --check passed.
  • ./mvnw -o spotless:apply -P with-integration-tests completed successfully for all modules.
  • Tests were added but were not executed locally. CI will run the unit and integration tests.

Track borrowed table sessions until release and close them when the pool closes. Make wrapper close idempotent and preserve pool capacity on failure paths.
Return the borrowed session when sortTablet succeeds or throws, and remove an unused connection-exception cleanup overload.
@codecov

codecov Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.73006% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.24%. Comparing base (c6e194d) to head (ea5036d).
⚠️ Report is 19 commits behind head on master.

Files with missing lines Patch % Lines
...va/org/apache/iotdb/session/SessionConnection.java 52.00% 12 Missing ⚠️
...ava/org/apache/iotdb/session/pool/SessionPool.java 90.76% 6 Missing ⚠️
...apache/iotdb/session/pool/TableSessionWrapper.java 94.87% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #19247      +/-   ##
============================================
+ Coverage     49.09%   49.24%   +0.14%     
- Complexity      715      717       +2     
============================================
  Files          5645     5648       +3     
  Lines        415676   416729    +1053     
  Branches      54871    55035     +164     
============================================
+ Hits         204080   205212    +1132     
+ Misses       211596   211517      -79     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Coordinate pool close with session borrowing and release to verify that sessions are not returned to a closed pool.

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lifecycle problems described in this PR are real. I checked the fixes with a fake Thrift server probe (a mock IClientRPCService.Iface behind a TThreadPoolServer, plus mocked Sessions), running the same probes against the merge base and against this head:

Scenario merge base this PR
pool.close() while a table session is borrowed session left open session closed
sql_dialect restore fails, maxSize=1 size=1 and empty queue, so the pool is exhausted for good capacity kept
tree-model result set closed after pool.close() session pushed into the closed pool's queue session closed
ITableSession.close() after a connection failure NPE returns normally

There is one new problem, though. pool.close() now waits for in-flight RPCs on borrowed table sessions, and it holds the pool monitor while it waits (see the comment on close()). On this head, with one table session running a statement that the server holds for about 4.5 s, close() took 4528 ms, and a concurrent getSession() was blocked on the monitor for 1.5 s before it failed with Session pool is closed. On the merge base, close() returned in 0 ms. Because connectionTimeoutInMs defaults to 0 (no socket read timeout), one stuck RPC can hang close(), and every other borrow and return, indefinitely. I think this should be fixed before merging.

The other comments are smaller:

  • The tree-model query path still has the close race that this PR fixes for table sessions.
  • The wrapper still throws an NPE if it is reused after a connection failure.
  • Two of the capacity fixes have no test.
  • A few nits, marked as such.

CI: the only failing IT job is IoTDBLoadTsFileClusterIT.testRestartedDataNodeIsSyncedWithTheLoadedFile (tree model, 1C3D), which looks unrelated. The new unit tests and table-model ITs ran and passed.

// do nothing
LOGGER.warn(CLOSE_THE_SESSION_FAILED, e);
}
closeSession(session);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Borrowed table sessions are now tracked in occupied, so this loop closes sessions that other threads may be using at that moment, and it holds the pool monitor while doing so. Session.close() calls SessionConnection.close(), which sends closeSession through the synchronized client proxy (SynchronizedHandler locks on the client). The call therefore waits until the other thread's in-flight RPC returns.

Probe: a fake server, a table pool with maxSize=2, and one borrowed session running a statement that the server holds for about 4.5 s.

  • This PR: close() returned after 4528 ms. A concurrent getSession() waited on the pool monitor for 1.5 s and then got Session pool is closed.
  • Merge base: close() returned in 0 ms, because borrowed table sessions were not touched here.

connectionTimeoutInMs defaults to 0, which means no socket read timeout. One stuck RPC can therefore hang close() indefinitely. Everything that needs the monitor hangs with it: getSession(), putBack(), releaseSession(), and TableSessionWrapper.close() (through isPoolClosed()).

Suggestion: under the lock, set closed = true and take and clear snapshots of queue and occupied. Close the snapshots after leaving the monitor. For sessions that are still in use, consider closing the transport directly instead of queueing a closeSession RPC behind the in-flight call. SessionConnection already has forceCloseTransport().

closeSession(session);
throw new IoTDBConnectionException(SESSION_POOL_IS_CLOSED);
}
occupied.put(session, session);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixes the borrow/close race for table sessions, but the tree-model query path still has it. The 12 query methods call occupy(session) after the RPC returns, and occupy() does not check closed. If close() runs while the query RPC is in flight, the session goes into occupied after close() has already cleared it.

Probe: executeQueryStatement blocked inside the RPC, then pool.close(), then the RPC returns. The wrapper is still returned, session.close() was called 0 times, and currentOccupiedSize() is 1 on the closed pool. The new releaseSession() does close the session once the caller closes the result set. If the caller relies on pool.close() for cleanup, for example by abandoning result sets at shutdown, the connection stays open.

Could occupy() do the same check under the lock as this code, closing the session and throwing if the pool is closed? Another option is to register every borrowed session in getSession() itself, under the lock that already checks closed. That would also remove the window between the two lock acquisitions here.

@Override
public void insert(Tablet tablet) throws StatementExecutionException, IoTDBConnectionException {
try {
session.insertRelationalTablet(tablet);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

releaseOnConnectionFailure() and close() both set session to null. If the same wrapper is called again, this line dereferences null and throws NullPointerException instead of an IoTDBConnectionException. Both executeQueryStatement overloads and executeNonQueryStatement behave the same way.

Probe: the first executeNonQueryStatement fails with IoTDBConnectionException. The second throws NullPointerException: Cannot invoke "org.apache.iotdb.session.Session.executeNonQueryStatement(String)" because "this.session" is null. The merge base behaves the same, so this is not a regression. Still, a caller that retries on the same ITableSession after a connection error will hit it, and this PR already makes close() safe after a connection failure. A guard at the top of these methods that throws IoTDBConnectionException, with the message as an i18n constant, would make the behavior consistent.

}

sessionPool.putBack(session);
sessionPool.releaseSession(sessionToRelease, reusable);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two of the fixes listed in the description have no test:

  1. Restoring sql_dialect/database fails: the session is closed and pool capacity is kept (this line with reusable = false).
  2. A connection failure in insert/execute*: releaseOnConnectionFailure() closes the session and replenishes the pool.

If either path went back to a bare session.close(), every test in this PR would still pass.

I checked (1) with a probe: maxSize=1, a getSqlDialect() that differs from the pool's, and a restore that throws StatementExecutionException. The merge base ends with size=1 and an empty queue, so the pool is exhausted for good. This PR ends with size=0: in the probe the replacement could not connect, so the slot was freed. In SessionPoolTest, mocking a failing executeNonQueryStatement and then asserting size, currentOccupiedSize() and currentAvailableSize() would protect both paths.

}
}

protected void cleanSessionAndMayThrowConnectionException(ISession session) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SessionPool is public and not final, and this protected overload has existed since 2.0.1-beta. Removing it breaks any subclass that calls it. Recompiling against the new jar fails, and running an old build against the new jar throws NoSuchMethodError. If compatibility matters here, a @Deprecated overload that delegates to discardSession(session) would keep those subclasses working. Also note that releaseSession and isPoolClosed become new protected API.

}
if (reusable && !closed) {
queue.push(session);
this.notifyAll();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: only one session is pushed here, but notifyAll() wakes every waiting borrower. putBack() and tryConstructNewSession() use notify(). With many waiters, each release wakes all of them; all but one get null from poll(), and each of those calls notify() again. notify() is enough here and matches putBack().

ISession session, int times, IoTDBConnectionException e) throws IoTDBConnectionException {
closeSession(session);
tryConstructNewSession();
if (!releaseSession(session, false)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: every caller of this method is a tree-model operation, and its session is not yet in occupied when the exception happens. releaseSession(session, false) therefore always returns false here, and discardSession() does the actual work. discardSession() also duplicates the close-then-replenish tail of releaseSession(). A single private discard(session) that removes the session from occupied, closes it, and replenishes if the pool is open could serve both callers, so the two paths cannot drift apart.

session = null;
}

if (sessionToRelease == null) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: closed = true is always set together with session = null. sessionToRelease can only be null here if closed was already true, so this branch is unreachable. Either use session == null as the closed state, or keep closed and drop this branch. The this.closed = false in the constructor is also redundant.

sessionPool.releaseSession(sessionToRelease, reusable);
}

private IoTDBConnectionException releaseOnConnectionFailure(IoTDBConnectionException exception) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit (stale comments): the class Javadoc still says the wrapper calls cleanSessionAndMayThrowConnectionException on connection errors, but that overload is gone and this method releases through releaseSession. Likewise, the comment on occupied in SessionPool (// for session whose resultSet is not released.) is no longer accurate, because borrowed table sessions are tracked there now. That also changes what currentOccupiedSize() reports.

// the borrowed session is registered in occupied.
while (!pool.isPoolClosed()) {
try {
pool.wait(5_000);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: both race tests depend on implementation details, and if those change, the failures hide the cause.

  • This pool.wait() works only because getSession() calls poll() while holding the pool monitor. If poll() moves out of the lock, pool.wait() throws IllegalMonitorStateException inside the borrow. The test then fails on assertTrue(e.getCause() instanceof IoTDBConnectionException) with no message, so the real exception is never shown.
  • testTableSessionReleaseRacingWithPoolClose depends on getDatabase() being evaluated before sessionPool.database != null. If someone swaps that condition, which is a natural micro-optimization, the latch never counts down. The test then fails at assertTrue(aboutToRelease.await(...)) after 5 s, again with no message.

Close borrowed sessions without holding the pool monitor, guard reuse after connection failure, and cover capacity recovery paths.
Prevent tree-model result queries from registering a session in occupied after the pool has closed.
Reconcile pooled session synchronization and result-set detach fixes from master with occupied tracking, force close, and tree query lease checks.

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. I re-ran my earlier probes against a6ee777, and every point from my previous review is addressed:

  • close() returns in about 1 ms with an RPC in flight (it took 4528 ms before).
  • A tree-model query that finishes after close() now fails with Session pool is closed.
  • Reusing a wrapper after a connection failure throws IoTDBConnectionException.
  • Capacity is kept when restoring the dialect fails.
  • sortTablet no longer borrows a session.

Switching to forceClose() introduces one new problem: an RPC that is in flight when the pool closes reconnects silently. Details are in the inline comment.

}

/** Closes the underlying transports without waiting for a close-session RPC. */
public synchronized void forceClose() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

forceClose() closes the transports and marks this Session as closed, but it does not stop a thread that is in the middle of an RPC on the session. That thread gets a TException and enters SessionConnection.callWithRetry() (or callWithRetryAndReconnect() for queries). reconnect() does not check whether the session has been closed. It opens a new transport, sends openSession, attaches the new connection back to this Session with setDefaultSessionConnection(this), and re-sends the statement.

I probed this on a6ee777, using a fake Thrift server that counts live connections and a table pool with one borrowed session running a statement that the server holds:

  • pool.close() returns in 1 ms, as intended.
  • The in-flight statement then returns success, but the server executed it twice: once on the old connection and once after the reconnect.
  • A new statement sent on the same ITableSession after pool.close() also succeeds.
  • After ts.close(), and even after an explicit Session.close(), the server still has 1 live connection and has received no closeSession. isClosed is already true, so close() and forceClose() return immediately, while defaultSessionConnection points at the new, open connection.

So every statement that is in flight at shutdown is sent a second time and leaves behind a connection, and a server-side session, that nothing closes. The new forceClose tests use a mocked SessionConnection/Session, so they don't cover this path.

Possible fixes:

  • Add a closed flag to SessionConnection that forceCloseTransport() sets. callWithRetry(), callWithRetryAndReconnect() and reconnect() then check it and throw IoTDBConnectionException instead of reconnecting.
  • Or, after reconnect() succeeds, close the new connection immediately if the owning Session has already been closed. For the reconnecting thread to see that reliably, isClosed needs to be volatile or read under the session's lock.

@shuwenwei
shuwenwei force-pushed the fix/table-session-pool-lifecycle branch from 1eeb127 to e85e1e9 Compare November 18, 2026 07:53
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants