[c] Support vector search materialized read in C FFI bindings#537
Draft
JunRuiLee wants to merge 7 commits into
Draft
[c] Support vector search materialized read in C FFI bindings#537JunRuiLee wants to merge 7 commits into
JunRuiLee wants to merge 7 commits into
Conversation
JunRuiLee
marked this pull request as ready for review
July 18, 2026 03:42
JunRuiLee
marked this pull request as draft
July 18, 2026 07:39
…e tests for pk-vector read
…e_read Extend VectorSearchBuilder::execute_read() to materialize rows for data-evolution (global-index) vector search, matching the primary-key path, so both storage types return projected user columns plus a unified score column. - Unify the user-visible score column to __paimon_search_score (constant SEARCH_SCORE_COLUMN), matching the engine metadata column name. Reject _ROW_ID, _PKEY_VECTOR_POSITION, __paimon_search_score, and the legacy _PKEY_VECTOR_SCORE alias as read projections. - Add attach_scores_by_row_id: join materialized rows to their (rank, score) by global _ROW_ID, reorder to the search's best-first rank (not re-sorted by score), append the score column, and strip _ROW_ID. Fail loud on missing / null / wrong-type / unknown _ROW_ID or a row count that disagrees with the scored result; never emit NaN or silently drop a row. - Add the data-evolution branch to execute_read: run the scored search, build the row-range read exposing _ROW_ID, materialize, and attach scores. Validate ids fit i64 (via to_row_ranges) before building the score map. A filter stays unsupported here and fails loud inside the scored search; the primary-key filter path is unchanged. Covers rank-order fidelity, row-id alignment, and the fail-loud branches with unit and end-to-end tests.
Add a C FFI vector-search surface over the Rust VectorSearchBuilder, so C/C++/Go consumers can run a vector search and read back the matched rows. - Builder handle over owned state (table + accumulated query params) plus setters for vector column, query vector, limit, options, and an optional scalar filter. The query vector and options are copied at the setter; the filter is consumed, matching the read-builder contract. - A single terminal, paimon_vector_search_builder_execute_read, runs the search and returns the existing streaming record-batch reader (paimon_result_record_batch_reader); consumers drain it with the existing reader-next / arrow-batch-free / reader-free functions. The entry is storage-agnostic and does no table-type routing — the core decides. Errors (including a filter on the append / data-evolution path) surface through the result's error field; an empty result is a normal end-of-stream reader. - Compile-time ABI signature guards pin every exported symbol. Integration tests compare the C stream against an independent Rust reference across primary-key and data-evolution tables, cover a primary-key filter that excludes a neighbor, the data-evolution filter error, setter validation, and the empty end-of-stream case.
JunRuiLee
force-pushed
the
feat/c-ffi-vector-search
branch
from
July 19, 2026 11:21
a2e6b72 to
7097120
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Part of #514. Stacked on #544 (primary-key vector physical-coordinate read) — review/merge after #544 lands; until then this PR's diff includes #544's commits.
Expose vector search to C/C++/Go consumers (e.g. Doris) via
paimon-c. After #544, primary-key vector search produces materialized rows (physical(file, position)coordinates, no global row ids), so the unified user-facing shape across storage types is materialized rows + a score column — matching Java Spark, where both primary-key and data-evolution reads surface rows plus a__paimon_search_scoremetadata column. This PR wires that through:execute_read()gains a data-evolution branch, and a single C entry returns the existing Arrow-stream reader.Brief change log
feat(table):VectorSearchBuilder::execute_read()now materializes rows for both primary-key and data-evolution vector search. Unified user-visible score column__paimon_search_score(was PK-only_PKEY_VECTOR_SCORE, kept as a rejected projection alias). Data-evolution rows are joined to their scores by global_ROW_ID, reordered to the search's best-first rank (not re-sorted by score), then_ROW_IDis stripped; any missing / null / wrong-type / unknown id, or a row count that disagrees with the scored result, fails loud (never a NaN score or a silently dropped row). A filter on the data-evolution path fails loud (parity is a follow-up); the primary-key filter path is unchanged.feat(c): newbindings/c/src/vector_search.rs— a builder handle over owned state, setters (with_vector_column/with_query_vector/with_limit/with_options/with_filter), and a single terminalpaimon_vector_search_builder_execute_readreturningpaimon_result_record_batch_reader(reuses the existing Arrow C Data Interface reader; consumers drain it with the existingpaimon_record_batch_reader_next/paimon_arrow_batch_free/paimon_record_batch_reader_free). The entry is storage-agnostic — C does no table-type routing; the core decides. Errors surface via the result'serrorfield; an empty result is a normal end-of-stream reader. Compile-time ABI signature guards pin every exported symbol.Tests
_ROW_ID, count mismatch); reserved-projection rejection.execute_read()reference across primary-key and data-evolution tables; a primary-key filter that excludes a neighbor; the data-evolution filterInvalidInputerror; setter validation; and the empty end-of-stream case.cargo test -p paimonandcargo test -p paimon-cgreen;cargo build -p paimon-datafusionclean (cross-crateSendgate);cargo clippy -D warningsandcargo fmt --checkclean.API and Format
No on-disk format change. New C ABI symbols only (add-only, each guarded by a compile-time signature assertion). This replaces the earlier scored (row-ids + scores) C entry, which was never released.
Documentation
Doc comments on the new FFI functions; no user-facing docs change.