Repository navigation
branch-4.1: [fix](nereids) Fix deep copy of scans with virtual columns #69399 - #69563
Open
airborne12 wants to merge 2 commits into
Open
airborne12 wants to merge 2 commits into
airborne12 wants to merge 2 commits into
Conversation
### What problem does this PR solve? Related PR: apache#69399 Problem Summary: Backport apache#69399 to branch-4.1. A CTE that orders by score() or by an ANN distance with a LIMIT, or that has MATCH in its select list and is inlined after rewrite, failed with "A expression contains slot not from children". LogicalPlanDeepCopier gave each virtual column two ids, copied them in list order, kept the old slot in the output of the copied scan and left the score and ANN order keys on the old slots. The copier now maps every virtual column to its new id before copying their expressions and remaps the order keys, and LogicalOlapScan.withVirtualColumns replaces the old virtual slots in the output. The branch has no table streams, so the change to LogicalOlapTableStreamScan is dropped. withVirtualColumns keeps the constructor call of the branch, and LogicalPlanDeepCopierTest and CTEInlineTest keep the tests of the branch and add the tests of apache#69399. ### Release note Fix "A expression contains slot not from children" for a CTE that orders by score() or by an ANN distance with a LIMIT, or that has MATCH in its select list and is inlined after rewrite. ### Check List (For Author) - Test: Includes the unit and regression tests of apache#69399; results are recorded in the pull request. - Behavior changed: No. - Does this need documentation: No. Source-PR: apache#69399 Source-Head: 5be9ff5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4 of 5 tasks
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
run buildall |
Contributor
FE Regression Coverage ReportIncrement line coverage |
### What problem does this PR solve? Related PR: apache#69399 Problem Summary: CI on branch-4.1 draws enable_common_expr_pushdown at random for each session. With it off, MATCH is not evaluated with the inverted index in the scan, so no BM25 score is collected and score() returns 0. test_cte_inline_virtual_column got score 0.0 for its TopN. Like the score suites of the branch, the suite sets enable_common_expr_pushdown = true. ### Release note None ### Check List (For Author) - Test: Regression test. With use_fuzzy_session_variable = true and the pull request number of CI, the suite failed in sessions that drew enable_common_expr_pushdown = false; with the setting pinned, four runs passed. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
run buildall |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Member
Author
|
run p0 |
Contributor
FE Regression Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Related PR: #69399
Problem Summary: Backport #69399 to
branch-4.1.A CTE fails to plan with "A expression contains slot not from children" when the optimizer inlines it after one of its scans got a virtual column: a CTE that orders by
score()or by an ANN distance with a LIMIT, or that has MATCH in its select list and is inlined after rewrite.LogicalPlanDeepCopiergave each virtual column two ids, copied them in list order, kept the old slot in the output of the copied scan and left the score and ANN order keys on the old slots. The copier now maps every virtual column to its new id before copying their expressions and remaps the order keys, andLogicalOlapScan.withVirtualColumnsreplaces the old virtual slots in the output.Branch adaptations
The branch has no table streams, so the change to
LogicalOlapTableStreamScanis dropped.withVirtualColumnskeeps the constructor call of the branch, andLogicalPlanDeepCopierTestandCTEInlineTestkeep the tests of the branch and add the tests of #69399.test_cte_inline_virtual_columnsetsenable_common_expr_pushdown = true, as the score suites of the branch do: CI on branch-4.1 draws it at random for each session, and without it MATCH is not evaluated with the index in the scan andscore()is 0.Release note
Fix "A expression contains slot not from children" for a CTE that orders by
score()or by an ANN distance with a LIMIT, or that has MATCH in its select list and is inlined after rewrite.Check List (For Author)
Validation
Validated together with the other two backports (#69399, #69419 and #69292 on
branch-4.1at 3f911d0), in a local ASAN build:LogicalPlanDeepCopierTest,CTEInlineTest,InferPredicateByReplaceTest,InferPredicatesTest,UnequalPredicateInferTest,BindScoreToOlapScanTest,SearchExpressionTest,TopNRuntimeFilterTestandMaterializedViewUtilsTest(228 tests), and FE Checkstyle../build.sh --beand./build.sh --fe.SegmentIteratorScoreCommonExprTest,SegmentIteratorApplyIndexExprTestandVirtualSlotRefTest(31 tests); clang-format 16 on the changed BE files.score(), includingtest_cte_inline_virtual_column,infer_match_predicate,test_score_filter_usageandtest_score_filter_matrix. The framework regenerated the.outoftest_score_filter_matrixon this branch: it drops the 46 SNII cases and keeps the 94 V2 and V3 cases of master unchanged. The other.outfiles are unchanged.use_fuzzy_session_variable = trueand its pull request number), the score suites failed in sessions that drewenable_common_expr_pushdown = false; with the setting pinned, four runs oftest_score_filter_usage,test_score_filter_matrixandtest_cte_inline_virtual_column, and four ofinfer_match_predicate, passed.🤖 Generated with Claude Code