Skip to content

feat(table): align primary-key vector read to physical-coordinate model#544

Open
JunRuiLee wants to merge 5 commits into
apache:mainfrom
JunRuiLee:feat/pk-vector-physical-coords
Open

feat(table): align primary-key vector read to physical-coordinate model#544
JunRuiLee wants to merge 5 commits into
apache:mainfrom
JunRuiLee:feat/pk-vector-physical-coords

Conversation

@JunRuiLee

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: part of #514

Rust's primary-key (PK) vector read assumed every PK-vector data file carries a first_row_id, using it to turn physical (file, row position) coordinates into global row ids. But Java never assigns first_row_id to PK-table data files — row-tracking (the only thing that assigns it) is forbidden on PK tables. Java's PK vector search works entirely in physical coordinates and deliberately does not produce global row ids: PrimaryKeyScoredResult.results() throws, and the physical position reader selects rows by file-local positions via batch.selection(rowPositions).

As a result, on a real Java-written PK-vector table (no first_row_id) both the scored path and the materialized read fail loud in Rust. The existing tests only passed because they pinned first_row_id == 0, making first_row_id + position == position.

This PR aligns the Rust PK vector read to Java's physical-coordinate model.

Brief change log

  • PK execute_scored() now fails loud when the query targets a PK-vector column (mirrors Java PrimaryKeyScoredResult.results() throwing): the PK path produces physical positions, not global row ids, so callers must use execute_read(). Removed the global-id helper that computed first_row_id + position.
  • Reworked the PK materialization read to select rows by file-local position (mirroring Java batch.selection(rowPositions)) instead of round-tripping through a global _ROW_ID/first_row_id. Added a local-selection read entry on DataFileReader that feeds file-local ranges straight to the format reader's row-selection, bypassing the global→local conversion.
  • _PKEY_VECTOR_POSITION/_PKEY_VECTOR_SCORE are emitted from an explicit cursor over the DV-filtered local selection, advanced by each batch's actual returned row count. The predicate-free guard is preserved (a row-filtering predicate would desync the cursor), and the local-selection reader rejects predicate readers self-consistently.
  • Dropped the first_row_id requirement from the residual-filter position collection; surviving rows' file-local positions are recovered from their ordinal in the unfiltered scan.

The DE/append (global-index) vector path and the shared SearchResult { row_ids, scores } shape are untouched — global row ids are legitimate there.

Tests

  • Cross-language proof: a real PK-vector table written by Java's production ivf-flat indexer is committed as opaque binaries and read back via execute_read() (pk_vector_java_fixture_test.rs). Its data files have no first_row_id — exactly the condition that broke the old code. Provenance (Java commit, generator command, schema/config, query + analytic top-k, checksum) is recorded in the test header. This becomes a permanent regression guard.
  • execute_read with first_row_id = None selects the correct file-local positions; with first_row_id = Some(100) (non-position-aligned) still returns the correct rows — catches any accidental reuse of the global row-range semantics.
  • Position↔row alignment across a deletion vector and multiple batches.
  • PK execute_scored() now returns an error; DE execute_scored/to_row_ranges tests stay green.

cargo test -p paimon passes; cargo clippy -p paimon --lib --tests -- -D warnings and cargo fmt --check clean.

API and Format

No storage-format change. Behavior change: PK execute_scored()/execute() now return a clear error instead of (incorrect) results computed under the first_row_id == 0 assumption — a corrective change on a path that was already broken for real Java-written data. No public physical-coordinate result type is added.

Documentation

No documentation changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant