Skip to content

branch-4.1: [fix](nereids) Fix deep copy of scans with virtual columns #69399 - #69563

Open
airborne12 wants to merge 2 commits into
apache:branch-4.1from
airborne12:backport/69399-branch-4.1
Open

airborne12 wants to merge 2 commits into
apache:branch-4.1from
airborne12:backport/69399-branch-4.1

Conversation

@airborne12

@airborne12 airborne12 commented Oct 24, 2026 •

Copy link
Copy Markdown
Member

What problem does this PR solve?

Related PR: #69399

Problem Summary: Backport #69399 to branch-4.1.

A CTE fails to plan with "A expression contains slot not from children" when the optimizer inlines it after one of its scans got a virtual column: a CTE that orders by score() or by an ANN distance with a LIMIT, or that has MATCH in its select list and is inlined after rewrite. LogicalPlanDeepCopier gave each virtual column two ids, copied them in list order, kept the old slot in the output of the copied scan and left the score and ANN order keys on the old slots. The copier now maps every virtual column to its new id before copying their expressions and remaps the order keys, and LogicalOlapScan.withVirtualColumns replaces the old virtual slots in the output.

Branch adaptations

The branch has no table streams, so the change to LogicalOlapTableStreamScan is dropped. withVirtualColumns keeps the constructor call of the branch, and LogicalPlanDeepCopierTest and CTEInlineTest keep the tests of the branch and add the tests of #69399. test_cte_inline_virtual_column sets enable_common_expr_pushdown = true, as the score suites of the branch do: CI on branch-4.1 draws it at random for each session, and without it MATCH is not evaluated with the index in the scan and score() is 0.

Release note

Fix "A expression contains slot not from children" for a CTE that orders by score() or by an ANN distance with a LIMIT, or that has MATCH in its select list and is inlined after rewrite.

Check List (For Author)

  • Test
    • Regression test
    • Unit Test
    • Manual test
  • Behavior changed:
    • No.
  • Does this need documentation?
    • No.

Validation

Validated together with the other two backports (#69399, #69419 and #69292 on branch-4.1 at 3f911d0), in a local ASAN build:

  • PASS: FE unit tests LogicalPlanDeepCopierTest, CTEInlineTest, InferPredicateByReplaceTest, InferPredicatesTest, UnequalPredicateInferTest, BindScoreToOlapScanTest, SearchExpressionTest, TopNRuntimeFilterTest and MaterializedViewUtilsTest (228 tests), and FE Checkstyle.
  • PASS: ./build.sh --be and ./build.sh --fe.
  • PASS: BE unit tests SegmentIteratorScoreCommonExprTest, SegmentIteratorApplyIndexExprTest and VirtualSlotRefTest (31 tests); clang-format 16 on the changed BE files.
  • PASS: the 16 regression suites of the branch that use score(), including test_cte_inline_virtual_column, infer_match_predicate, test_score_filter_usage and test_score_filter_matrix. The framework regenerated the .out of test_score_filter_matrix on this branch: it drops the 46 SNII cases and keeps the 94 V2 and V3 cases of master unchanged. The other .out files are unchanged.
  • PASS: with the fuzzy session variables of CI (use_fuzzy_session_variable = true and its pull request number), the score suites failed in sessions that drew enable_common_expr_pushdown = false; with the setting pinned, four runs of test_score_filter_usage, test_score_filter_matrix and test_cte_inline_virtual_column, and four of infer_match_predicate, passed.

🤖 Generated with Claude Code

### What problem does this PR solve?

Related PR: apache#69399

Problem Summary: Backport apache#69399 to branch-4.1. A CTE that orders by
score() or by an ANN distance with a LIMIT, or that has MATCH in its
select list and is inlined after rewrite, failed with "A expression
contains slot not from children". LogicalPlanDeepCopier gave each virtual
column two ids, copied them in list order, kept the old slot in the
output of the copied scan and left the score and ANN order keys on the
old slots. The copier now maps every virtual column to its new id before
copying their expressions and remaps the order keys, and
LogicalOlapScan.withVirtualColumns replaces the old virtual slots in the
output.

The branch has no table streams, so the change to
LogicalOlapTableStreamScan is dropped. withVirtualColumns keeps the
constructor call of the branch, and LogicalPlanDeepCopierTest and
CTEInlineTest keep the tests of the branch and add the tests of apache#69399.

### Release note

Fix "A expression contains slot not from children" for a CTE that orders
by score() or by an ANN distance with a LIMIT, or that has MATCH in its
select list and is inlined after rewrite.

### Check List (For Author)

- Test: Includes the unit and regression tests of apache#69399; results are
  recorded in the pull request.
- Behavior changed: No.
- Does this need documentation: No.

Source-PR: apache#69399
Source-Head: 5be9ff5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@airborne12

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 44.44% (12/27) 🎉
Increment coverage report
Complete coverage report

### What problem does this PR solve?

Related PR: apache#69399

Problem Summary: CI on branch-4.1 draws enable_common_expr_pushdown at
random for each session. With it off, MATCH is not evaluated with the
inverted index in the scan, so no BM25 score is collected and score()
returns 0. test_cte_inline_virtual_column got score 0.0 for its
TopN. Like the score suites of the branch, the suite sets
enable_common_expr_pushdown = true.

### Release note

None

### Check List (For Author)

- Test: Regression test. With use_fuzzy_session_variable = true and
the pull request number of CI, the suite failed in sessions that
drew enable_common_expr_pushdown = false; with the setting pinned,
four runs passed.
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@airborne12

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 44.44% (12/27) 🎉
Increment coverage report
Complete coverage report

@airborne12

Copy link
Copy Markdown
Member Author

run p0

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 44.44% (12/27) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants