Skip to content

Reject text indexes on multidimensional array columns - #116895

Open
JasonLi-cn wants to merge 33 commits into
ClickHouse:masterfrom
JasonLi-cn:text-index-reject-nested-array
Open

Reject text indexes on multidimensional array columns#116895
JasonLi-cn wants to merge 33 commits into
ClickHouse:masterfrom
JasonLi-cn:text-index-reject-nested-array

Conversation

@JasonLi-cn

@JasonLi-cn JasonLi-cn commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Text indexes currently accept multidimensional arrays whose base type is String or FixedString, although the runtime only handles scalar values and one-dimensional arrays correctly. Array(Array(String)) fails later when the index is built, while Array(Array(FixedString)) can concatenate adjacent fixed-width values and build incorrect tokens.

Reject these definitions during CREATE TABLE, ALTER TABLE ... ADD INDEX, replicated database DDL, and user-supplied full-definition ATTACH TABLE. Existing metadata remains loadable during startup, short ATTACH, restore, and recovery. Legacy indexes retain their previous runtime behavior until users remove them with ALTER TABLE ... DROP INDEX.

The regression tests cover String and FixedString, scalar and one-dimensional supported controls, full-definition ATTACH, stored-metadata replay, replicated database validation, and the preserved legacy behavior.

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Reject new text indexes on multidimensional arrays with a String or FixedString base type, which previously failed while building the index or could generate incorrect tokens. Existing table metadata remains loadable so users can remove legacy indexes with ALTER TABLE ... DROP INDEX.


Workflow [PR]
Sync PR [sync-upstream/pr/116895]

@CLAassistant

CLAassistant commented Aug 28, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@JasonLi-cn
JasonLi-cn force-pushed the text-index-reject-nested-array branch 2 times, most recently from d9833a7 to dcc4b49 Compare August 28, 2026 15:30
@m-selmi m-selmi added the can be tested Allows running workflows for external contributors label Sep 1, 2026
@clickhouse-gh

clickhouse-gh Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Workflow [PR], commit [ff01ca4]

Summary:


AI Review

Summary

This PR now consistently rejects new text indexes on multidimensional String/FixedString array expressions while keeping previously committed metadata loadable through full-definition ATTACH, DatabaseReplicated replay, and ReplicatedMergeTree metadata replay. I did not find a remaining correctness or rolling-upgrade gap in the current diff that warrants a new inline review comment.

Missing context / blind spots
  • ⚠️ I could not exercise the #if CLICKHOUSE_CLOUD SharedDatabaseCatalog replay branches from this OSS checkout. A Cloud-side regression test or internal CI coverage would close that gap.
Final Verdict

✅ No new findings in the current diff.

LLVM Coverage Report

Measured on commit ff01ca4.

Metric Baseline Current Δ
Lines 89.00% 89.00% +0.00%
Functions 91.80% 91.80% +0.00%
Branches 81.50% 81.40% -0.10%

Changed lines: Changed C/C++ lines covered: 45/48 (93.75%) · Uncovered code

Full report · Diff report

@clickhouse-gh clickhouse-gh Bot added pr-bugfix Pull request with bugfix, not backported by default comp-text-index Full-text search: inverted/text index in MergeTree (tokenizers, posting lists, GIN-style index bu... labels Sep 1, 2026
Reject new `text` indexes on multidimensional arrays with a `String` or `FixedString` base type during `CREATE TABLE`, `ALTER TABLE ... ADD INDEX`, replicated database DDL, and full-definition `ATTACH TABLE`.

Keep startup, short `ATTACH`, restore, and recovery compatible with legacy metadata. Legacy indexes retain their existing runtime behavior until users remove them with `ALTER TABLE ... DROP INDEX`.
@JasonLi-cn
JasonLi-cn force-pushed the text-index-reject-nested-array branch from dcc4b49 to db4be29 Compare September 7, 2026 03:02
Comment thread src/Storages/MergeTree/registerStorageMergeTree.cpp Outdated
@clickhouse-gh

clickhouse-gh Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Build profile diff (arm_release)

Comparing ff01ca4e4 with master 0d04f2bc2 (stripped binary size, per-symbol sizes and ThinLTO time; compile times per translation unit against the most recent warmup build that recompiled it).

✅ No significant changes.

Binary sizes

programs/clickhouse-stripped: smaller than the master baseline by the known offset between the two builds, so the difference is not shown. A delta that differs from the offset by more than 50% of it is shown, in either direction.

The official master build is compiled with -g and a pull request build is not, and XRay counts debug instructions towards its instrumentation threshold, so master instruments thousands of functions more and its binary is ~0.4% larger no matter what the pull request does.

Compile time of recompiled translation units

10 translation units recompiled, 88 s compile time in total, 10 of them have a recent master baseline.

Job report

@JasonLi-cn

Copy link
Copy Markdown
Contributor Author

🕵 The remaining failures are unrelated to this change and are being addressed separately:

Comment thread src/Storages/MergeTree/MergeTreeIndexText.cpp
@JasonLi-cn

Copy link
Copy Markdown
Contributor Author

🕵 The Build (arm_tidy) failure is unrelated: it is the duplicate DataTypeLowCardinality include in MergeTreeIndexConditionText.cpp, which is already addressed by #118680 (also independently by #118661).

Comment thread src/Storages/MergeTree/registerStorageMergeTree.cpp Outdated
Comment thread src/Storages/StorageReplicatedMergeTree.cpp
Comment thread tests/integration/test_text_index_legacy_attach/test.py
Comment thread tests/integration/test_text_index_legacy_attach/test.py
Comment thread src/Storages/MergeTree/registerStorageMergeTree.cpp Outdated
Comment thread src/Storages/MergeTree/MergeTreeData.cpp
JasonLi-cn and others added 4 commits September 11, 2026 12:47
…s CI flavors

The test reference expected the per-replica DDL status row from
`CREATE TABLE` in a `Replicated` database, but the DBReplicated flavor
sets `distributed_ddl_output_mode = none` (tests/config/users.d/database_replicated.xml),
which suppresses that row, so the test failed there with empty output.

Force `--distributed_ddl_output_mode=none` on every client call in the
test and empty the reference, matching the convention used by other
`Replicated` database tests.

Co-authored-by: Cursor <cursoragent@cursor.com>
@JasonLi-cn

Copy link
Copy Markdown
Contributor Author

🕵 @groeneai, two CI failures on the latest run look unrelated to this change (which only touches text index DDL validation and its tests):

  1. Upgrade check (amd_release)Logical error: Virtual row has different type for CAST(__table1.json.b, 'String'_String) ... in setVirtualRow (STID: 1637-309b). This is failing master-wide since 2026-09-10 16:22 UTC: 21 occurrences across ~15 unrelated PRs plus master itself (ad8622edd0fb, Stress test (amd_tsan)). A candidate fix is already open: Fix stale virtual row conversion kept when ORDER BY contains constants #119195
    Report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=116895&sha=latest&name_0=PR&name_1=Upgrade%20check%20%28amd_release%29

  2. AST fuzzer (amd_release, oracle)TLP Aggregate oracle mismatch on SELECT count() FROM tab_both STREAM BOUNDED UNORDERED CURSOR .... The fuzzer run involved no text index at all (tab_both is plain MergeTree with minmax indexes), and the query uses the experimental streaming-cursor feature. Could you investigate: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=116895&sha=latest&name_0=PR&name_1=AST%20fuzzer%20%28amd_release%2C%20oracle%29 and provide a fix in a separate PR? If a fix is already in progress, please link it here.

The third failure (05137_text_index_replicated_database_alter in the DBReplicated flavor) was caused by this PR's test itself — the reference expected the per-replica DDL status row, which that flavor suppresses via distributed_ddl_output_mode = none. Fixed in 6f386a3d323 by forcing the setting in the test.

@JasonLi-cn

Copy link
Copy Markdown
Contributor Author

🕵 Filed a tracking issue for the AST fuzzer streaming-cursor mismatch: #119420

@groeneai

Copy link
Copy Markdown
Collaborator

Thanks. A fix for the AST fuzzer failure is already open: #117188, "Do not answer count() from part metadata for a STREAM read". Your #119420 is the same signature, so it can be linked to that PR.

It is a real wrong result rather than oracle non-determinism. With your DDL, cursor and predicate on master, the bare reference count() returns the whole-table count and ignores the CURSOR bound (15 instead of 12), while every WHERE-carrying arm reads through the streaming reader and is correctly bounded, repeatably. Only the bare reference arm is eligible for the implicit min-max/count shortcut, which is answered from part metadata. Measurements and the mechanism: #119420 (comment)

For the setVirtualRow failure I have #119385 open alongside @ vdimir's #119195. Neither is merged yet, so there is nothing for this branch to pick up from master.

@JasonLi-cn

Copy link
Copy Markdown
Contributor Author

🕵 Merged current master (0d04f2bc274) into this branch so CI can pick up #119400.

The remaining 03100_lwu_deletes_4_index failure (expected read_rows 8, got 0) is unrelated to text index DDL validation. CI minimized it to --min_bytes_for_wide_part 0 --patch_parts_version v1, which is the Wide + v1 patch-part read_rows = 0 accounting bug from #114562. That fix is already on master (merged 2026-09-12); CIDB shows no further failures of this test after that date, including on master itself.

The public three-dot diff is unchanged (text-index validation and tests only). CH Inc sync was still reporting conflicts on the previous head; I do not have access to ClickHouse/clickhouse-private from this checkout, so that check will need a follow-up on the sync PR after this merge lands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

can be tested Allows running workflows for external contributors comp-text-index Full-text search: inverted/text index in MergeTree (tokenizers, posting lists, GIN-style index bu... pr-bugfix Pull request with bugfix, not backported by default

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants