Do not leave a Filesystem database pointing at a removed directory, which makes the server refuse to start - #119618
Merged
Conversation
`05153_database_filesystem_cache_read_on_file_grant` creates a `Filesystem`
database over a directory under `user_files` and then removes both, but the
two cleanups are not atomic and they fail independently: `DROP DATABASE` is a
call into the server and fails whenever the server is gone -- which is routine
under a stress run, where the server is killed and restarted while the tests
run -- while `rm -rd` is local and always succeeds.
The database's metadata then outlives the directory it points at, and every
later server start aborts while loading metadata:
Code: 36. DB::Exception: Path does not exist:
/var/lib/clickhouse/user_files/d_05153_test_..._21520: while loading
database `db_05153_test_..._23401` from file metadata/....sql.
(BAD_ARGUMENTS)
so the server never comes back and the job fails with `Cannot start
clickhouse-server`, as seen on
https://github.com/ClickHouse/ClickHouse/actions/runs/34678356221/job/103529922983
(`Stress test (arm_asan_ubsan, s3)`).
Remove the directory only once the database that points at it is really gone.
A dead server makes `EXISTS DATABASE` produce nothing, the directory is kept,
and the leftover is a harmless empty directory instead of a server that cannot
start. This mirrors what `05057_file_rename_after_processing_write_grant`
already does for the same hazard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
Workflow [PR], commit [1bb1ed5] Summary: ✅ AI ReviewSummaryThis PR changes the cleanup in Missing context / blind spots
Final Verdict✅ No blocking issues found in the current diff. |
Member
|
Must backport this to fix CI |
robot-ch-test-poll
added
the
pr-must-backport-synced
robot-clickhouse-ci-1
added
the
pr-synced-to-cloud
This was referenced Sep 13, 2026
robot-ch-test-poll
added
the
pr-backports-created
clickhouse-gh Bot
pushed a commit
that referenced
this pull request
Sep 13, 2026
…ng at a removed directory, which makes the server refuse to start
clickhouse-gh Bot
added a commit
that referenced
this pull request
Sep 13, 2026
Backport #119618 to 26.8: Do not leave a `Filesystem` database pointing at a removed directory, which makes the server refuse to start
clickhouse-gh Bot
added a commit
that referenced
this pull request
Sep 13, 2026
Backport #119618 to 26.7: Do not leave a `Filesystem` database pointing at a removed directory, which makes the server refuse to start
PedroTadim
added a commit
that referenced
this pull request
Sep 13, 2026
Backport #119618 to 26.3: Do not leave a `Filesystem` database pointing at a removed directory, which makes the server refuse to start
clickhouse-gh Bot
added a commit
that referenced
this pull request
Sep 13, 2026
Backport #119618 to 26.6: Do not leave a `Filesystem` database pointing at a removed directory, which makes the server refuse to start
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
05153_database_filesystem_cache_read_on_file_grantcreates aFilesystemdatabase over a directory underuser_filesand removes both at the end. The two cleanups are not atomic and they fail independently:DROP DATABASEis a call into the server and fails whenever the server is gone — routine under a stress run, where the server is killed and restarted while the tests run — whilerm -rdis local and always succeeds.The database's metadata then outlives the directory it points at, and every later server start aborts while loading metadata:
so the server never comes back and the job fails with
Cannot start clickhouse-server. Seen onStress test (arm_asan_ubsan, s3): https://github.com/ClickHouse/ClickHouse/actions/runs/34678356221/job/103529922983 (report: https://s3.amazonaws.com/clickhouse-test-reports/praktika.html?PR=116647&sha=fca3eb858cfcfc30c835375b45f976add5188544&name_0=PR&name_1=Stress%20test%20(arm_asan_ubsan,%20s3) , PR #116647 — the failure is unrelated to that pull request's changes).The fix removes the directory only once the database that points at it is really gone. A dead server makes
EXISTS DATABASEproduce nothing, the directory is kept, and the leftover is a harmless empty directory instead of a server that cannot start. This mirrors what05057_file_rename_after_processing_write_grantalready does for the same hazard.The other
Filesystemdatabase tests were checked and are not affected:02722_database_filesystemand04653_database_filesystem_concurrent_resolvecreate the database over theuser_filesroot, which no test removes.Verified on a local server: the test passes and leaves no directory behind, and
EXISTS DATABASEanswers1for a live database,0for a dropped one and nothing at all when the server is down.Related: #116647
Documentation entry for user-facing changes
Changelog category (leave one):
🤖 Generated with Claude Code
Workflow [PR]
Sync PR [sync-upstream/pr/119618]
Version info
26.9.1.1380(included in26.9and later)26.8.3.106,26.7.7.94,26.6.5.121,26.3.33.70