[core] Support source table FileIO for BLOB descriptors#8693
Open
LsomeYeah wants to merge 1 commit into
Open
Conversation
LsomeYeah
force-pushed
the
codex/blob-descriptor-source-table
branch
from
July 16, 2026 09:05
2ea960b to
0ae8567
Compare
Contributor
|
Thanks for this PR! |
leaves12138
reviewed
Jul 16, 2026
| private static final long serialVersionUID = -8477284718943635074L; | ||
|
|
||
| @Nullable private final CatalogContext context; | ||
| @Nullable private final FileIO fileIO; |
Contributor
There was a problem hiding this comment.
1、FileIO should be transient
2、It seems strange to hold both fileIO and readers
Contributor
There was a problem hiding this comment.
You can just overwrite newReader() to just put your special fileIO in
leaves12138
reviewed
Jul 16, 2026
| Set<Integer> blobFields) { | ||
| this.row = row; | ||
| this.uriReaderFactory = new UriReaderFactory(catalogContext); | ||
| this.uriReaderFactory = uriReaderFactory; |
Contributor
There was a problem hiding this comment.
If FlinkRowWrapper with uriReaderFactory(fileIO), then it can't be successfully serialize and deserialize. FileIO is not serializable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
When a source table exposes BLOB values as descriptors, a downstream write must reopen the referenced objects before writing them into the target table. Rebuilding
FileIOfrom the target table catalog context can miss source data permissions, especially for token-based catalogs.This PR adds the optional
blob-descriptor.source-tableoption. When configured, Paimon loads that table from the same catalog and uses its serializableFileIOto materialize descriptor-backed BLOBs. The existing target-context behavior remains unchanged when the option is absent.Changes
FileIOfor non-HTTP descriptor URIs.mapToInternalRowAPI and route the new factory path through a private helper.blob-descriptor.source-tableis used by a target without a catalog loader, including REST external tables.Compatibility and limitations
blob-descriptor.source-tabletakes precedence over otherblob-descriptor.*FileIO options.CatalogContextbehavior is unchanged.Tests
BlobDescriptorReaderFactoryTestandUriReaderFactoryTestBlobTableITCaseBlobTestBasePaimonBatchWriteTest