Repository navigation
Conversation
- Add bvecs_writer.py: write/append/count for .bvecs (int8 per-vector
with int32 dim header), the INT8 equivalent of fvecs_writer.py
- readers.py: ParquetEmbeddingReader gains output_dtype param ('float32'
or 'int8'); build_reader forwards it so parquet INT8 columns are read
as np.int8 without any float cast
- fvecs_remove_zeros.py: detect .bvecs suffix, read/write int8, cast to
float32 only for norm computation
- fvecs_normalize.py: detect .bvecs suffix; copy file through unchanged
(INT8 vectors cannot be meaningfully normalized in-place); emits
expected log lines so processing.py metadata parsing works
- fvecs_deduplicator.py: add _vec_nbytes/_vec_fmt helpers; all reader,
writer, merge, and presorted-dedup paths detect .bvecs and use d bytes
per vector with 'b' struct format instead of 4*d/'f'
- fvecs_split.py: read_dim_and_count gains is_bvecs param; main detects
suffix and uses record_size = 4+dim instead of 4+4*dim
- config.py: add SOURCE_INT8 flag (from dataset config, default False);
all intermediate working-file paths use .bvecs extension when true
- processing.py: import append_bvecs/count_bvecs; extraction uses
append_bvecs when SOURCE_INT8; count_output_file handles .bvecs;
final artifact rename uses correct extension
Add config for wikipedia-2023-11 multilingual-v3 INT8 dataset (1M base, 10k query, emb_int8 column, SOURCE_INT8=True, output to /home/ubuntu/dataset_cache/int8-datasets/processed/)
- add read_bvecs(): reads int8 records (4+dim bytes each), upcasts to float32 for FAISS - add write_bvecs(): clips float32 back to int8 for processed output files - add _write_vectors(): dispatches to write_fvecs or write_bvecs by suffix - read_vectors(): detect .bvecs extension and call read_bvecs - all processed_base_out/processed_query_out writes go through _write_vectors so .bvecs paths are handled correctly
…-mixed-bread-large
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds end-to-end support for processing INT8 quantized vector datasets into
.bvecsformat. This includes passthrough handling across normalization, deduplication, zero-removal, and splitting stages,.bvecssupport in ground-truth k-NN calculations, and download/processing configurations for several INT8 datasets.Key Changes
1. INT8 and
.bvecsSupport in the PipelineNew
bvecs_writer.py.bvecsbinary format:d) stored asint32int8/uint8Pipeline Passthrough Support
processing.pyandconfig.pyto support:.bvecsoutputsfvecs_normalize.pyfvecs_remove_zeros.pyfvecs_deduplicator.pyfvecs_split.pyGround Truth Stage
knn_utils.pyto load.bvecsdatasets and compute ground-truth nearest neighbors.Readers
readers.pyto correctly handle INT8 columns and formats.2. New Dataset Download and Processing Configurations
MixedBread Large INT8
mich_embeddings_int8_mixed_bread_large_download.pymich_embeddings_int8_mixed_bread_large_proc.pyWikipedia 2023-11 EN MXBAI INT8 (1M)
wikipedia_2023_11_en_mxbai_int8_download.pywikipedia_2023_11_en_mxbai_int8_1m_proc.pyWiki Multilingual v3 INT8 (1M)
wiki_multilingual_v3_int8_1m_proc.py3. Dependencies and Tooling
requirements.txtlisting required third-party dependencies:huggingface_hubpyarrowscikit-learnnumpy