Add validate_group_by_all_key_types to gate the GROUP BY ALL key type validation - #119868
Add validate_group_by_all_key_types to gate the GROUP BY ALL key type validation#119868groeneai wants to merge 1 commit into
validate_group_by_all_key_types to gate the GROUP BY ALL key type validation#119868Conversation
…type validation `GROUP BY ALL` adds the `SELECT` expressions as grouping keys in `expandGroupByAll`, which runs after `resolveGroupByNode` has already done its tuple unwrapping and key type validation for an explicit `GROUP BY`. The keys `GROUP BY ALL` expands into therefore went through neither, until ClickHouse#110206 re-ran both on them: `expandTuplesInList`, which fixes ClickHouse#83433, and a `validateGroupByKeyType` loop. That loop is an acceptance rule, and it changed behavior: a `Variant`/`Dynamic` grouping key written as `GROUP BY ALL`, typically an untyped JSON subpath, is rejected with `ILLEGAL_COLUMN` from 26.7 onward where earlier versions accepted it. The tightening is intended, so this keeps it as the default and only makes it reachable by `compatibility`. Reported in ClickHouse#119852 with a measured upgrade break (3 of 20 sampled chReplay instances) and an identical result hash once the key was permitted, i.e. the old acceptance computed correct answers rather than wrong ones; without a gate an upgrade breaks those queries with no way to keep them running short of rewriting them or enabling `allow_suspicious_types_in_group_by`, which also loosens the explicit `GROUP BY` and `ORDER BY` checks. The setting is read at the two call sites rather than inside `DB::validateGroupByKeyType`: that helper is shared with the explicit `GROUP BY`, `GROUP BY GROUPING SETS`, the window `PARTITION BY` and both old-analyzer sites, none of which changed behavior, so gating the helper would newly loosen all of them. Two call sites, not one, because `GROUP BY ALL` reaches the validation two ways. Normally the late `was_group_by_all` block validates. But when `scope.group_by_use_nulls` is set, which needs `group_by_use_nulls` plus a `WITH ROLLUP`/`CUBE`/`GROUPING SETS` modifier, `GROUP BY ALL` is expanded early and `expandGroupByAll` clears the flag, so that block is skipped and `resolveGroupByNode` validates the expanded keys instead. Hence the ALL-ness is captured once before any expansion and threaded in as a policy; an explicit `GROUP BY` passes `true` and is bit-for-bit unchanged. The `nullable_group_by_keys` collection sharing those loops stays outside the guard: it implements `group_by_use_nulls` promotion, which the setting must not affect, and moving it inside would reintroduce the bug fixed in b3edb0b. The history entry goes in the 26.7 block, not 26.8. ClickHouse#110206 (9d67075) merged 2026-07-16, inside the 26.7 development window, and `gh api .../compare/v26.7.1.1315-stable...9d67075` reports `behind`, so the rejection first shipped in 26.7; recording it under 26.8 would tell `compatibility` that 26.7 accepted such a key, which it did not. The second entry in the current 26.9 block is required by `ci/jobs/check_style.py::check_settings_changes_history`, which demands every added setting under the current version block whenever another `src/` file changes. Closes: ClickHouse#119852 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Internal second-model review: adjudication log (click to expand)Pre-publication review by an independent model (engine: codex), plus my own cold review of the
Two documented deviations from the internal implementation plan were re-derived and accepted on the Severity: ❌ blocker / Session id: cron:clickhouse-review-slot-8:20260913-234400 |
Pre-PR validation gate (click to expand)
Session id: cron:clickhouse-impl-slot-6:20260913-233005 |
|
Workflow [PR], commit [98b8363] Summary: ❌
AI ReviewSummaryThis PR adds Missing context / blind spots
Final VerdictNo actionable findings. LLVM Coverage Report |
Build profile diff (arm_release)Comparing ✅ No significant changes. Binary sizes
The official master build is compiled with Compile time of recompiled translation units22 translation units recompiled, 178 s compile time in total, 22 of them have a recent master baseline. Median compile-time ratio to the baselines is ×1.08 (machine-speed difference or a change affecting every TU); per-TU deltas below are relative to that ratio. |
CI finish ledger - 98b8363CI is complete on this head (344 check runs, aggregator finished 04:24:38Z). Every failure below has an owner: a fixing PR (mine or external), or a full-effort fix task whose fixing-PR link will be posted here when it opens.
Session id: cron:our-pr-ci-monitor:20260914-043100 |
|
Fixing PR for the |
Closes: #119852
Related: #110206
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
New setting
validate_group_by_all_key_types(defaulttrue) gates the key type validation thatGROUP BY ALLapplies to the grouping keys it expands theSELECTexpressions into. Since 26.7 it rejects aVariantorDynamicgrouping key, for example an untyped JSON subpath, that earlier versions accepted with the analyzer enabled (the default), with no way to keep such a query running across an upgrade. Settingcompatibilityto a version before 26.7, orvalidate_group_by_all_key_types = 0, restores the earlier behavior. An explicitGROUP BYis unaffected and still rejects such a key, as it did before 26.7. Closes #119852.Description
GROUP BY ALLadds theSELECTexpressions as grouping keys inexpandGroupByAll, afterresolveGroupByNodehas validated an explicitGROUP BY, so before #110206 the expanded keys were never type-checked. #110206 re-ran both the tuple expansion (fixing #83433) andvalidateGroupByKeyTypeon them. That tightening is intended and stays the default; what shipped without an escape hatch is the acceptance change, socompatibilitycould not keep an upgraded workload running. Requested by @ fm4v in #119852, who measured it on 3 of 20 sampled chReplay instances, with an identical result hash: the old acceptance computed correct answers.Only the validation is gated, keyed on the query having been written
GROUP BY ALL. The tuple expansion stays unconditional and the sharedvalidateGroupByKeyTypehelper is untouched, so explicitGROUP BY,GROUPING SETS, windowPARTITION BYand the old analyzer keep rejecting such keys as before. The gate covers both places these keys are validated: the latewas_group_by_allblock, andresolveGroupByNode, which validates them instead whengroup_by_use_nullsmeetsWITH ROLLUP/CUBE(expanded early there, flag already cleared).Two measured corrections to the issue's suggested shape:
SettingsChangesHistoryentry goes in the 26.7 block. The requestedcompatibility = '26.7'arm is in the test as a negative arm;'26.6'is the restore arm. Moving it to 26.8 is a one-line change if you prefer.Default
truereproduces current behavior exactly, so no existing query changes.Distributedis deliberately out of scope: a shard receives the expanded keys as an explicitGROUP BY, which 26.6 rejected too, so there is nothing to restore there. I can ask for a backport label after merge.Evidence for the two corrections, and the validation
1. #110206 (
9d67075) is in 26.7.gh api repos/ClickHouse/ClickHouse/compare/<tag>...9d67075 --jq .status:v26.6.5.120-stable...9d67075divergedv26.7.1.1315-stable...9d67075behindv26.8.1.2041-lts...9d67075behindIts test file is present at
ref=v26.7.1.1315-stableand absent atref=v26.6.5.120-stable. It merged 2026-07-16, inside the 26.7 window (26.7 opened 2026-06-23, 26.8 opened 2026-07-22). The issue compared 26.6.1.2103 against 26.8.1.189, so 26.7 was never tested. Recording it under 26.8 would tellcompatibilitythat 26.7 accepted such a key, which it did not.2. The 26.9 entry.
check_style.py::check_settings_changes_historydemands every added setting under the current version block whenever anothersrc/file changes, so a 26.7-only entry fails Style check. 149 settings already sit in more than one block. No 26.8 entry: nothing changed in 26.8.Validation. The issue's reproducer throws
ILLEGAL_COLUMNon the parent commit and returns its 5 rows with the setting off orcompatibility = '26.6';compatibilityresolves the setting to0for 26.4/26.6 and1for 26.7/26.8/26.9. Verified forVariantand forDynamicnested inTuple/Array/Map, with the explicit-GROUP BYform still rejected in every case.04602_group_by_all_suspicious_typesgains 10 arms, including the reported JSON-subpath shape; four were confirmed load-bearing by breaking the fix four ways (gating the tuple expansion, gating the shared helper, gating only the late block, moving the history entry to 26.8) and checking the predicted arm went red each time. 100/100 green over 50 randomized runs of04602plus04518_group_by_all_tuple_order_by.Workflow [PR]
Sync PR [sync-upstream/pr/119868]