Skip to content

[core] Support source table FileIO for BLOB descriptors#8693

Open
LsomeYeah wants to merge 1 commit into
apache:masterfrom
LsomeYeah:codex/blob-descriptor-source-table
Open

[core] Support source table FileIO for BLOB descriptors#8693
LsomeYeah wants to merge 1 commit into
apache:masterfrom
LsomeYeah:codex/blob-descriptor-source-table

Conversation

@LsomeYeah

@LsomeYeah LsomeYeah commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Purpose

When a source table exposes BLOB values as descriptors, a downstream write must reopen the referenced objects before writing them into the target table. Rebuilding FileIO from the target table catalog context can miss source data permissions, especially for token-based catalogs.

This PR adds the optional blob-descriptor.source-table option. When configured, Paimon loads that table from the same catalog and uses its serializable FileIO to materialize descriptor-backed BLOBs. The existing target-context behavior remains unchanged when the option is absent.

Changes

  • Resolve the configured source table once and use its FileIO for non-HTTP descriptor URIs.
  • Pre-initialize lazy source credentials before distributing the reader factory, avoiding per-task REST token requests.
  • Resolve the reader factory outside distributed Spark tasks, including postpone compaction.
  • Preserve the existing public Flink mapToInternalRow API and route the new factory path through a private helper.
  • Fail fast when blob-descriptor.source-table is used by a target without a catalog loader, including REST external tables.
  • Document same-catalog and branch usage, option precedence, and the unsupported REST external target case.

Compatibility and limitations

  • The source table must belong to the same catalog; a branch suffix is supported.
  • REST external target tables are not supported because they do not provide a catalog loader.
  • blob-descriptor.source-table takes precedence over other blob-descriptor.* FileIO options.
  • Without this option, the existing target CatalogContext behavior is unchanged.

Tests

  • BlobDescriptorReaderFactoryTest and UriReaderFactoryTest
  • Flink SQL end-to-end BLOB materialization in BlobTableITCase
  • Spark V1 and V2 SQL end-to-end BLOB materialization in BlobTestBase
  • Spark V2 writer-factory regression test in PaimonBatchWriteTest
  • Spark postpone compaction test verifying that the source table is not resolved in tasks
  • Flink null-overload source compatibility and row-wrapper compatibility tests
  • Configuration documentation completeness test
  • Spark 4 compilation with JDK 17
  • Flink 2 compilation with JDK 11

@LsomeYeah
LsomeYeah force-pushed the codex/blob-descriptor-source-table branch from 2ea960b to 0ae8567 Compare July 16, 2026 09:05
@steFaiz

steFaiz commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Thanks for this PR!
Does the target table store the blobs from source table as blob format or blob descriptors?
If as blob format, I think we can use blob-descriptor.XXX to specify options for CatalogContext as in #7231
If as blob descriptors, I think blob-view is designed to solve this.

private static final long serialVersionUID = -8477284718943635074L;

@Nullable private final CatalogContext context;
@Nullable private final FileIO fileIO;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1、FileIO should be transient
2、It seems strange to hold both fileIO and readers

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can just overwrite newReader() to just put your special fileIO in

Set<Integer> blobFields) {
this.row = row;
this.uriReaderFactory = new UriReaderFactory(catalogContext);
this.uriReaderFactory = uriReaderFactory;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If FlinkRowWrapper with uriReaderFactory(fileIO), then it can't be successfully serialize and deserialize. FileIO is not serializable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants