Skip to content

branch-4.0: [fix](paimon) preserve partition metadata across splits#65583

Merged
morningman merged 2 commits into
apache:branch-4.0from
suxiaogang223:codex/fix-paimon-partition-metadata-40
Jul 15, 2026
Merged

branch-4.0: [fix](paimon) preserve partition metadata across splits#65583
morningman merged 2 commits into
apache:branch-4.0from
suxiaogang223:codex/fix-paimon-partition-metadata-40

Conversation

@suxiaogang223

Copy link
Copy Markdown
Member

What problem does this PR solve?

Paimon partition metadata was generated only when runtime partition pruning was enabled. Some count/native/JNI splits therefore reached BE without stable partition values. BE also reused a partition-prune block across splits but inserted replacement columns, corrupting the block layout.

Changes

  • Always attach ordered Paimon partition metadata to every data split.
  • Generate aligned columns_from_path_keys, columns_from_path, and columns_from_path_is_null arrays in FE.
  • Pass explicit null markers from FileScanner to Parquet/ORC readers.
  • Preserve NULL, empty string, and literal \N semantics.
  • Replace partition columns in the reusable prune block instead of inserting new columns.
  • Preserve the partition-key case behavior already present on the latest branch-4.0 baseline.
  • Add FE/BE unit coverage and a three-partition static IN regression case.

This is a focused backport of the relevant #62821 behavior adapted to the pre-#62306 reader architecture.

Validation

  • clang-format 16
  • git diff --check
  • FE checkstyle for touched Java files
  • Full FE/BE tests will run in CI; local FE UT compilation is blocked by missing generated Thrift Java classes in this worktree.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 60.00% (21/35) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 53.16% (42/79) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.53% (19572/36562)
Line Coverage 36.60% (183171/500435)
Region Coverage 33.18% (142467/429426)
Branch Coverage 34.02% (61563/180981)

@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 60.00% (21/35) 🎉
Increment coverage report
Complete coverage report

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Jul 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@morningman
morningman merged commit 7047fb2 into apache:branch-4.0 Jul 15, 2026
26 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants