Repository navigation
Conversation
Reuse ordered output buffers and read-name groups, incorporate safe mate and formatting fast paths, and stream only-depth from reusable records. Use published Seqair 0.10.1 without the experimental fork or unchecked vector operations. Preserve depth admission and reject unsupported BAM long-CIGAR pileups explicitly. Amp-Thread-ID: https://ampcode.com/threads/T-01a1ca76-7f80-72d6-ac92-b1b4e5bacdcf Co-authored-by: Amp <amp@ampcode.com>
Replace the long-CIGAR rejection guard with a counting regression and include optional Seqair support in release binaries while retaining HTSlib as the default. Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-01a1ca76-7f80-72d6-ac92-b1b4e5bacdcf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge prerequisites
Do not merge until the long-CIGAR fix lands in upstream Seqair. This remains a draft for review.
Cargo.lockbefore the Perbase release; Git-only dependencies cannot be published to crates.io.Scope
Bring the useful application-level results from the Goose/Rust experiments into Perbase. The candidate temporarily pins Seqair and seqair-types to the long-CIGAR fix, not our experimental performance branch. HTSlib remains the default. Build with
--features seqairand select--seqairon either depth command to opt in. The release workflow now includes the feature in downloadable binaries, uses Rust 1.98.1, and checks that both depth commands expose--seqairbefore uploading.-Cbudget constrains queues rather than reducing counting to one worker.Compatibility and limitations
Seqair's generic depth cap does not match HTSlib's whole-read admission at boundaries; the adapter preserves Perbase's existing rule instead. The comparison tests include low and zero caps, ties, filters, and all nine mate strategies.
The optional backend requires Rust 1.98.1 or newer and an external FASTA for CRAM, including embedded-reference CRAM. The pinned fix expands BAM long CIGARs from
CGbefore pileup counting. The former Perbase rejection guard is removed, and its regression now checks the actual base and insertion counts for a 65,536-operation BAM. Both depth commands support these records.Validation
--seqairhelp checks and the Seqair CLI differential suite. The tag-triggered publishing workflow was not run; no binaries have been released, and the macOS release build has not been exercised in this update.cargo +1.98.1 test --locked: 132 unit tests and one doctest passed.cargo +1.98.1 test --locked --features seqair: 137 unit tests and one doctest passed.cargo fmt --checkandgit diff --checkpassed. Clippy completed; existing warnings in examples/older tests and dependency future-compatibility warnings remain.1M1Ioperations: each tested position has depth 1, an A count of 1, and an insertion count of 1, rather than a reference skip from the placeholder.Performance
Original measurements of this PR's first build versus a clean build of released Perbase 1.4.1. These predate the long-CIGAR pin and are not the experimental fork's R7 numbers. The table is retained as the original matched comparison, not mixed with newer measurements.
Chr20 BAM on the same computer,
-t 32 -c 1000000, default queue budgets, uncompressed TSV written to a regular file (no fsync). One warmup followed by two measured runs for each variant/workload, run sequentially in rotating order without concurrent builds or tests. All 27 outputs matched the corresponding workload's SHA-256. These are preliminary two-run measurements, not confidence intervals.base-depth -monly-depthonly-depth -mThe application changes improve both backends. Seqair gives another approximately 2.71× speedup over the optimized HTSlib base-depth path here; only-depth is essentially tied between the two new backends.
base-depth -monly-depthonly-depth -mRSS is the maximum over the two timed runs, not an average. Seqair's base-depth speed comes with more memory than the PR's HTSlib path because it retains each active region's reads.
Samples and build provenance
Seconds in measured order:
[49.47, 49.38]; PR HTSlib[19.55, 20.20]; PR Seqair[7.33, 7.35].[9.78, 10.35]; PR HTSlib[4.22, 4.13]; PR Seqair[4.18, 4.10].[10.19, 10.16]; PR HTSlib[4.16, 4.14]; PR Seqair[4.11, 4.13].Both source revisions built with Rust 1.98.1, their release profile,
RUSTFLAGS='-C target-cpu=native',CC=clang, andCFLAGS='-O3 -march=native'. Candidate built with--features seqair; both PR columns use that same executable, changing only--seqair. Baseline built from clean 1.4.1 with no features. Both commands used--lockedand their respective Cargo.lock files.a3527928f449355c693275c95fdf2712efaa6279ed57e461652b758d78ff6804.136131baf86623e9d55110318e691878ade0ce98704f6612a57b9ed076044060.b10b1722946bea9a8b952bf416579b94ba46c390b86a5b863fd3965f850c1eab.b4e1e41f38b69f5ae99c2a9b5fa652773dde07df41f4467572a6a2bc8de93ed7.825dd615bcba2d4e841e46e775a7b4a9ab75d146ddfb15f8b10a1956e5d20012.24556d4290684b96765e16bb8b0750f6cd8c1766438475d910224ce07fd61060.Historical timing recheck
The exact saved PR executable above, verified by SHA-256 and not rebuilt, subsequently measured 5.99 s for mate-fixed base-depth (samples 5.99, 6.03, 5.90), rather than its original 7.34 s. A newer unchanged control measured 5.88 s (5.88, 6.00, 5.87) in the same session. Each had one warmup and three measured runs in alternating order, with the same BAM, 32 threads, 1 Mb chunks, and full TSV output. All outputs matched the recorded checksum. Summed CPU medians were 74.06 and 74.54 seconds respectively.
This rules out a persistent slowdown in the old executable; it does not establish why the earlier environment was slower. Machine or I/O conditions remain hypotheses. Do not attribute the 7.34 → roughly 5.9 second difference to code changes, or substitute the new Seqair number into the old comparison against HTSlib and released Perbase.
Deferred Seqair work
The long-CIGAR correctness fix is now a pinned prerequisite, not deferred optimization work. Other library changes remain outside this application PR:
The experiment narrative contains the historical checkpoints. These library follow-ups are deliberately separate from this application PR.
AI disclosure
Prepared with Amp, including code and tests. This is a draft for Seth's full human review before merge, as required by CONTRIBUTING.md. No merge or release is requested by this PR.