OnDiskGraphIndexCompactor: accept vectors from outside the source graphs, write the result to a caller-provided output, and let the caller choose the output ordinals #775

Hi, I maintain ArcadeDB, a multi-model DBMS that embeds JVector (currently 4.0.2) for its LSM_VECTOR index. We're planning to move from a single graph that is rebuilt from scratch to a tiered design (sealed on-disk graphs plus one mutable on-heap graph, merged in the background, the way Lucene segments work): ArcadeData/arcadedb#5397. OnDiskGraphIndexCompactor is exactly the merge step we need, and we'd like to use it instead of writing our own. Three things in its current API stop us.

1. Vectors must be stored inside the source graphs

validateFeatures() requires FeatureId.INLINE_VECTORS on every source, and the merge reads every vector through OnDiskGraphIndex.View.getVectorInto(...): the merge store, the hub map, cross-source scoring and PQ retraining.

In ArcadeDB the full-precision vectors already live in the database's own pages, so to avoid storing every vector twice we persist graphs with InlineVectors of dimension 0 (topology only). This is the default, and it cut graph files to a fraction of their size. Those graphs pass the feature check but carry no vector data, so they can't be compacted.

Request: an overload that takes one vector source per input graph, indexed by that graph's local ordinal, for example:

public OnDiskGraphIndexCompactor(List<OnDiskGraphIndex> sources,
                                 List<RandomAccessVectorValues> sourceVectors,
                                 List<FixedBitSet> liveNodes,
                                 VectorSimilarityFunction similarityFunction,
                                 ForkJoinPool executor)

When sourceVectors is given, the INLINE_VECTORS requirement would be dropped and getVectorInto would read from the supplied vectors. The batched/bounded vector access added in 4.0.2 (#744) looks like the natural interface, since the merge already streams vectors in batches and our vectors come from paged storage rather than from heap.

2. Output only to a Path

compact(Path) and compact(Path, Path) write to a local file. The writer reopens it with FileChannel for positioned writes, truncates the scratch regions afterwards, and creates spill files next to it. ArcadeDB keeps graphs inside its own paged files: we write them through an IndexWriter implementation and read them back through a custom ReaderSupplier. Today we'd have to compact into a temporary file and copy it into pages, which needs the merged graph's size again in temporary disk space plus a full extra read and write.

Request: either of these would work for us:

• (a) keep the scratch and spill files under a caller-chosen working directory, but write the final compacted graph to a caller-provided IndexWriter (sequential, footer-based, which looks close to what the draft disk: host-owned streaming output contract #727 proposes);
• (b) accept a positioned-write sink (a RandomAccessWriter the caller implements) for the output, with the scratch regions handled separately.

3. Output ordinals chosen by the compactor

The compactor assigns the output ordinals itself, and the caller reads them back through ordinalMappers(). ArcadeDB keys its graph nodes by its own vector ids, and it looks a node up by binary search over an ordinal-to-id map, so the graph's ordinals must follow vector-id order. Today we'd have to read the compacted graph back and rewrite it once more in our order, a second full pass over the merged graph after the merge itself.

Request: let the caller pass, for each source, the output ordinal of each of its live nodes, for example:

compactor.withOutputOrdinals(List<int[]> outputOrdinals); // outputOrdinals.get(s)[node] = output ordinal

The merge itself would keep running on the compactor's own ordinals, since its cell join needs them; only the writer would translate on write: node records, neighbour lists, upper layers, the entry node and the PQ codes. Without the call the compactor numbers the output as it does today.

All three changes are additive: the existing constructors and compact(Path) keep their current behaviour.

Context

• At 1M vectors (128-d, EUCLIDEAN), each rebuild after churn currently takes about 4 minutes on 6 threads and about 17.6 minutes on one, and happens every 50k mutations. Replacing it with compaction of sealed tiers is the main motivation (measurements in LSM_VECTOR: incremental HNSW insertion with tiered graphs, to remove the full-rebuild ingestion bottleneck ArcadeData/arcadedb#5397).
• We'd pass our own ForkJoinPool (we avoid the common pool for engine work), which the current constructor already allows. Thanks for that.
• Related upstream work we're following: disk: host-owned streaming output contract #727 (streaming output), Remap heap graph ordinals on serialization #618 (heap ordinal remap), implement algorithm 5 for inplace repair and algorithm 6 to clean up … #648 (in-place delete repair).

We plan to implement all three changes and will send a PR soon for your review. If you already prefer one of the shapes above (in particular (a) vs (b) for the output, or a different vector-source interface than #744's batched access), let us know here and we'll follow it.