Stories by Vikram Bhaskaran on Medium

Packaging the World’s Video in Pure Rust

Vikram Bhaskaran — Fri, 03 Jul 2026 08:14:53 GMT

Building sheathe, an HLS/DASH/CMAF media packager, validated byte-for-byte against Shaka Packager

Every video you stream on YouTube, Netflix, or an OTT sports app arrives in pieces. A two-hour movie is not one file served in order — it is a manifest listing hundreds of small segments, each containing a few seconds of video and audio, joined by a description that tells the player how to stitch them back together. The tool that makes those pieces is called a packager, and production streaming has relied on exactly one open-source implementation for a decade: Google’s Shaka Packager.

This is the story of building a second one — in Rust, with no C or C++ anywhere in the dependency tree, validated against the original byte-for-byte.

What a Packager Actually Does

A packager takes a finished media file — say, a 1080p MP4 with H.264 video and AAC audio — and produces what the web needs to stream it:

CMAF segments. Common Media Application Format fragments, the ISO-BMFF-based chunks that every modern player understands. Each segment is a self-contained fMP4 file with an initialization header plus a sequence of moof+mdat fragments covering ~6 seconds.
A DASH manifest. An MPD (Media Presentation Description) XML document that lists every segment URL, its duration, its resolution, and its codec. A player reads the MPD, figures out which rendition matches the current network, and fetches segments.
An HLS playlist. A master .m3u8 listing available renditions plus per-track media playlists, with the same segment references. Older players and Apple devices use this path.

If the content is encrypted — and in production it always is — the packager also applies Common Encryption (CENC), embeds the DRM signalling, and writes the per-sample encryption metadata that a player’s Content Decryption Module needs to decrypt each frame.

That is the job. The reference implementation — Shaka Packager — does it in C++ with ~250,000 lines of code, covering a dozen input formats, four encryption schemes, live streaming, trick-play, and HLS packed-audio output. The goal of sheathe is functional parity on the subset that production VOD (Video on Demand) actually uses, done in pure Rust and differential-tested against Shaka at every step.

Why Build Another Packager?

There are excellent Rust projects that parse DASH manifests and HLS playlists. There was no mature Rust project that produces segments and manifests from video files — the write side is the hard side, and the ecosystem had a gap in the Delivery lane: probe the source, build the ladder, segment into CMAF, write the manifests.

sheathe fills that gap. It also fills a subtler one: the standard packager is a native binary written in C++, carrying all the memory-safety risk of parsing untrusted media containers at scale. A packager that does the same work in unsafe-free Rust turns a class of buffer-overflow CVEs into compile-time impossibilities. That matters the moment you point it at a file you didn't create.

Architecture: Nine Crates, Four Layers

sheathe is nine crates arranged in four layers, each crate with one responsibility:

Foundation   sheathe-core      Media model: streams, samples, timing, errors
             sheathe-mp4        ISO-BMFF box reader/writer, CMAF fragments
             sheathe-crypto     CENC encryption (all four schemes + DRM pssh)
             sheathe-ts         MPEG-2 transport stream demux (PAT/PMT/PES)
             sheathe-es         Raw elementary stream demux (Annex B, ADTS)

Pipeline     sheathe-dash       DASH MPD generation
             sheathe-hls        HLS master + media playlist generation

Application  sheathe-cli        The `sheathe` binary
             sheathe            CLI crate (thin wrapper over sheathe-cli)

The dependency graph points strictly down. A manifest crate composes foundation crates but never reaches sideways into another manifest crate. This is where Rust’s crate boundary earns its keep: the MP4 writer cannot accidentally call into the DASH MPD generator, because the compiler won’t let it. Nine shallow crates, each small enough to hold in your head and test in isolation.

The whole workspace is ~6,000 lines of Rust across 41 source files, with 34 tests that verify everything from AES-128-CTR NIST vectors to AVC codec-string derivation — and an integration harness that diffs output against Shaka Packager itself.

The Read Side: Demuxing MP4, MPEG-TS, and Elementary Streams

Before sheathe can write CMAF segments, it has to read the source. Three input paths are wired:

MP4 demux (sheathe-mp4). A dependency-free, cursor-based box reader traverses the ISO-BMFF hierarchy: moov → trak → mdia → minf → stbl. From the sample table (stts/ctts/stsc/stsz/stco/stss) it reconstructs every sample's decode time, composition offset, size, offset, and keyframe flag — the exact information the fragmenter needs to slice the stream into segments. The box reader is a simple state machine over a &[u8] reference; there is no allocation beyond the BoxIter stack, and no unsafe.

// Top-level box traversal: find every box at the current nesting level,
// yielding (type, body, next_offset) for each.
pub fn top_level(data: &[u8]) -> impl Iterator>> {
    // ... purely a Cursor over the byte slice
}

MPEG-TS demux (sheathe-ts). Transport streams are a different beast entirely — 188-byte packets carrying program-specific information (PAT/PMT) and packetized elementary stream (PES) payloads. sheathe-ts parses PAT to find the program, PMT to find the streams, PES headers to reassemble access units, and Annex B / ADTS parsers to extract H.264, HEVC, and AAC samples. It synthesizes avcC/hvcC/mp4a sample entries from the elementary stream headers, so the output segment's moov carries the decoder configuration the player needs — all without an external probe tool.

Elementary stream demux (sheathe-es). For raw .h264, .hevc, and .aac files, sheathe detects the format from the extension and content signature, splits the stream into access units (via Annex B start-code scanning for video, ADTS frame sync for audio), and feeds them to the same fragmenter. No container, no problem.

The Write Side: CMAF Segments from a Stream of Samples

Once the demuxer has produced a stream of Sample structs (data, decode timestamp, composition offset, duration, keyframe flag), the fragmenter decides where to cut. The policy is the same one Shaka uses: wait until the accumulated samples exceed the target segment duration (default 6 seconds), then cut at the next keyframe — because a segment that starts on a non-keyframe cannot be decoded independently, and DASH/HLS switching depends on clean boundaries.

pub fn push(&mut self, sample: Sample) -> Result<()> {
    let elapsed = sample.dts.saturating_sub(self.current_start);
    let may_cut = !self.policy.keyframes_only || sample.is_segment_boundary();
    if !self.current.is_empty() && may_cut && elapsed >= self.target_ticks {
        self.cut(sample.dts);
    }
    if self.current.is_empty() {
        self.current_start = sample.dts;
    }
    self.current.push(sample);
    Ok(())
}

Each segment is written as an fMP4 file: a styp box (segment type), a sidx box (segment index for seeking), a moof box (movie fragment — the sample table for this segment), and an mdat box (the actual sample data). The init segment carries ftyp + moov + mvex, describing the codec configuration once, at the start.

The box writer is a 75-line nestable builder with automatic size backpatching — no pre-computed table of contents. begin("moof"), write child boxes, end() — and the 32-bit size in the header is patched to the correct value. This is the same pattern Shaka uses; sheathe just does it in 1/10th the code.

The Hard Part: Common Encryption Done Correctly

Encryption is where most packagers get complex. CENC (ISO/IEC 23001–7) defines four protection schemes, and sheathe implements all of them:

SchemeCipherModeVideo PatternAudiocencAES-128-CTRFull-regionNAL-aware subsamplesWhole-samplecensAES-128-CTRPattern (1:9)Crypt 1 block, skip 9Whole-samplecbc1AES-128-CBCFull-regionPer-sample IV, block-alignedWhole-samplecbcsAES-128-CBCPattern (1:9)Crypt 1, skip 9, constant IVWhole-sample

The sheathe-crypto crate is a 457-line module built on the RustCrypto aes crate. The AES-128-CTR path runs a continuous keystream across protected byte ranges, skipping clear bytes and pattern-skipped blocks without advancing the counter. The AES-128-CBC path chains ciphertext blocks, resetting to the constant IV at each subsample boundary under pattern encryption. Both are validated against NIST SP 800-38A test vectors, and against Shaka Packager's output on real media.

Every scheme is ffmpeg decrypt+decode verified. That is: sheathe encrypts a sample, produces an fMP4 segment, and then ffmpeg -decryption_key ... -i segment.mp4 -f framemd5 confirms every video frame and every audio frame decodes to the same MD5 as the clear original. No "probably encrypted" — provably correct.

Multi-DRM pssh

Encrypting the bytes is half the job; the other half is telling the player’s DRM module how to decrypt them. Each DRM system gets a pssh box in the init segment, generated directly from the raw key (no key server required):

Common — a version-1 box listing the KID, used by W3C clear-key and urn:mpeg:dash:mp4protection.
Widevine — a version-0 box carrying a WidevinePsshData protobuf with the KID and protection-scheme fourcc.
PlayReady — a version-0 box wrapping a PlayReady Object with a WRMHEADER 4.0.0.0 (KID, key length, ALGID, and an AES-ECB checksum proving key possession).

Each box byte-matches Shaka Packager’s --protection_systems output. Not approximately — identically.

Key Rotation

For long content, rotating keys reduces the exposure window of any single key. sheathe supports --crypto-period-duration : each period derives a new content key by left-rotating the base key (Shaka's naive raw-key scheme), signalled per segment via seig sample groups (sbgp/sgpd), a zero-KID init tenc, and a per-period pssh in each moof. Every segment decrypts to the clear baseline under its derived key.

Manifest Generation: DASH and HLS

With segments written, the packager generates the manifests that players consume.

DASH. The MPD is an XML document in the static (type="static"), on-demand profile, using SegmentTemplate + SegmentTimeline — the same profile Shaka produces for VOD. Each representation gets a element with its codec string (RFC 6381), resolution, bandwidth, and a that lists each segment's exact duration. Equal-duration runs are collapsed with r= (repeat count), so a 2-hour movie doesn't produce 1,200 ~~elements.~~

The codec strings themselves are derived from raw box data — avcC → avc1.640028, hvcC → hvc1.1.6.L93.90, av1C → av01.0.04M.08, esds → mp4a.40.2 — in a pure-Rust, zero-dependency codec-string module with tests pinned against known outputs.

// avc1.PPCCLL from avcC
fn avc_string(fourcc: &[u8; 4], avcc: &[u8]) -> Option {
    let (profile, compat, level) = (*avcc.get(1)?, *avcc.get(2)?, *avcc.get(3)?);
    let prefix = std::str::from_utf8(fourcc).ok()?;
    Some(format!("{prefix}.{profile:02x}{compat:02x}{level:02x}"))
}

HLS. The master playlist lists video variants as #EXT-X-STREAM-INF entries with combined bandwidth and codecs (video + audio folded together, per the spec), bound to an #EXT-X-MEDIA audio rendition group. Media playlists carry #EXTINF per-segment durations, #EXT-X-MAP pointing to the init segment, and #EXT-X-KEY encryption signalling (SAMPLE-AES for cbcs, SAMPLE-AES-CTR for cenc) — the full HLS encrypted-fMP4 contract.

The Oracle: Differential Testing Against Shaka Packager

sheathe adopts the “revelo method”: implement in pure Rust, then differential-test against a reference oracle. For a packager, the oracle is Shaka Packager running the same input file through the same output configuration.

~~The oracle harness (just oracle input.mp4) runs both packagers on the same source, then diffs:~~

Segment counts. Do sheathe and Shaka produce the same number of media segments?
Canonical MPD. XML canonicalization (xmllint --c14n) strips formatting to compare structure — same representations, same timelines, same codec strings.
ffmpeg decode. Both outputs are fed through ffmpeg -f framemd5 to confirm the video renders identically.

This is the same methodology that keeps revelo’s 180+ parsers byte-identical to MediaInfoLib and viser’s convex hulls correct to the numerical method. Nothing ships until its bytes are verified against the oracle. If Shaka and sheathe disagree, the bug is in sheathe until proven otherwise.

What Makes This Genuinely Hard

~~Rust makes it safe; the domain makes it hard. The genuinely difficult parts of building a packager are worth naming:~~

NAL-aware subsample encryption is format-specific. AVC and HEVC samples cannot be encrypted blindly — the NAL unit headers and slice headers must stay clear for a player to parse the stream. The senc box carries a table of (clear_bytes, encrypted_bytes) per sample, and getting the boundary exactly right requires knowing the NAL unit length size from the avcC/hvcC configuration. Get it wrong, and the player produces a green screen or a crash.

The full CENC scheme matrix has subtle interactions. cbcs uses a constant IV across all samples; the other three derive per-sample IVs. cbc1 chains CBC across subsamples; cbcs resets to the constant IV at each subsample. Pattern encryption applies to video only — audio under cens/cbcs is whole-sample encrypted with Pattern::NONE. DASH-IF conformance requires saiz/saio auxiliary-information boxes with the saio offset backpatched to point at the senc data. Every one of these interactions earned its own differential test.

PlayReady pssh is deeply non-obvious. The PlayReady Object wraps a WRMHEADER with a KID that uses swapped GUID byte order (the first three fields of the UUID are written little-endian, not network order), and an AES-ECB checksum of the object. Building this from the raw key requires implementing a subset of the PlayReady binary format that no RFC describes — the oracle is the only spec.

The box model rewards getting it right the first time. ISO-BMFF boxes nest arbitrarily: moov contains trak, which contains mdia, which contains minf, which contains stbl, which contains stsd, which contains the actual sample entry with its decoder configuration. A missing box, a wrong offset, or an incorrect size computation produces a file that looks valid to a file browser and silently fails to play. The reader and writer together form a round-trip that is verified on every test asset.

The Result

sheathe is a pure-Rust HLS/DASH/CMAF media packager at ~6,000 lines, validated against Shaka Packager as the reference oracle. It demuxes MP4, MPEG-TS, and raw elementary streams; writes CMAF init and media segments with correct box structure and codec strings; applies all four CENC encryption schemes with multi-DRM pssh (Widevine, PlayReady, Common) and key rotation; and emits DASH MPDs and HLS playlists that match Shaka’s output.

# Package an MP4 into 6s CMAF segments with both DASH and HLS manifests.
sheathe package input.mp4 --out site/ --segment-duration 6 --dash --hls

# Encrypt with CENC cbcs (AES-CBC pattern) and Widevine+PlayReady DRM.
sheathe package input.mp4 --out site/ --dash --hls \
  --enc-scheme cbcs \
  --enc-key abcd1234...:deadbeef... \
  --protection-systems common,widevine,playready

# Probe an MPEG-TS file — no ffprobe required.
sheathe probe input.ts

# Install the binary from crates.io.
cargo install sheathe

sheathe doesn’t yet do live streaming, trick-play, or WebM input. Those are tracked on the roadmap, each gated behind an oracle validation checkpoint. What ships today is the complete VOD pipeline — the subset that covers ~90% of production packaging work — built in unsafe-free Rust, with no C dependencies, and every output differential-tested against the best open-source packager available.

One More Thing: The Pure-Rust Probe

A packager needs to know what it’s packaging. Resolution, frame rate, codec, bitrate — the numbers that determine how segments get sized and manifests get filled. The default way to get those numbers is to shell out to ffprobe.

sheathe doesn’t need to. Its MP4 demuxer reads sample-table metadata directly from the box hierarchy; its MPEG-TS demuxer reconstructs stream descriptions from PAT/PMT/PES headers; its elementary-stream path synthesizes sample entries from the Annex B/ADTS byte stream itself. The sheathe probe command returns the same information ffprobe does, with no external binary on the path.

For the full metadata picture — the kind that covers 177+ media formats with byte-identical output to MediaInfoLib — there’s revelo, another pure-Rust project that integrates as a drop-in probe engine. But for the subset a packager actually needs, sheathe is self-contained: the tool that segments video is also the tool that reads it. One binary, one dependency chain, one cargo install.

What I’d Want You to Take From This

~~If there’s a lesson worth carrying out of this project, it’s simpler than the topic implies:~~

Choose an oracle you can diff against. Shaka Packager is open source, battle-tested on billions of streams, and emits deterministic output. Every feature sheathe ships was validated by running both tools on the same input and comparing the result. Byte-identical output is the only claim worth making.
The crate boundary is a design tool. Nine crates, each with exactly one concern, separated by the compiler. The MP4 writer cannot reach into the DASH generator; the crypto crate knows nothing about the container format. Wide and flat, not deep and tangled.
The hardest bugs are interactions, not algorithms. AES-CTR is 40 lines. Getting the saiz offset to backpatch correctly after the senc is written — that's where the hours go. The differential harness catches these; no amount of code review does.
Packaging is infrastructure, not magic. A packager is a program that reads video, cuts it into pieces, and writes a description of the pieces. Doing it correctly is a lot of work; doing it in pure Rust is less work than you’d think, and the result is safer, simpler, and faster to install than the C++ alternative.

~~sheathe is open source at github.com/vbasky/sheathe, under MIT OR Apache-2.0.~~

The author is also building revelo (a pure-Rust MediaInfoLib port) and viser (a content-adaptive video encoding optimizer). The three projects form a suite: probe → ladder → package — the Delivery lane of a pure-Rust media ecosystem, end to end.

Packaging the World’s Video in Pure Rust

Vikram Bhaskaran — Sat, 27 Jun 2026 14:52:28 GMT

Building sheathe, an HLS/DASH/CMAF media packager, validated byte-for-byte against Shaka Packager
Every video you stream on YouTube, Netflix, or an OTT sports app arrives in pieces. A two-hour movie is not one file served in order — it is a manifest listing hundreds of small segments, each containing a few seconds of video and audio, joined by a description that tells the player how to stitch them back together. The tool that makes those pieces is called a packager, and production streaming has relied on exactly one open-source implementation for a decade: Google’s Shaka Packager.
This is the story of building a second one — in Rust, with no C or C++ anywhere in the dependency tree, validated against the original byte-for-byte.
What a Packager Actually Does
A packager takes a finished media file — say, a 1080p MP4 with H.264 video and AAC audio — and produces what the web needs to stream it:
CMAF segments. Common Media Application Format fragments, the ISO-BMFF-based chunks that every modern player understands. Each segment is a self-contained fMP4 file with an initialization header plus a sequence of moof+mdat fragments covering ~6 seconds.
A DASH manifest. An MPD (Media Presentation Description) XML document that lists every segment URL, its duration, its resolution, and its codec. A player reads the MPD, figures out which rendition matches the current network, and fetches segments.
An HLS playlist. A master .m3u8 listing available renditions plus per-track media playlists, with the same segment references. Older players and Apple devices use this path.
If the content is encrypted — and in production it always is — the packager also applies Common Encryption (CENC), embeds the DRM signalling, and writes the per-sample encryption metadata that a player’s Content Decryption Module needs to decrypt each frame.
That is the job. The reference implementation — Shaka Packager — does it in C++ with ~250,000 lines of code, covering a dozen input formats, four encryption schemes, live streaming, trick-play, and HLS packed-audio output. The goal of sheathe is functional parity on the subset that production VOD (Video on Demand) actually uses, done in pure Rust and differential-tested against Shaka at every step.
Why Build Another Packager?
There are excellent Rust projects that parse DASH manifests and HLS playlists. There was no mature Rust project that produces segments and manifests from video files — the write side is the hard side, and the ecosystem had a gap in the Delivery lane: probe the source, build the ladder, segment into CMAF, write the manifests.
sheathe fills that gap. It also fills a subtler one: the standard packager is a native binary written in C++, carrying all the memory-safety risk of parsing untrusted media containers at scale. A packager that does the same work in unsafe-free Rust turns a class of buffer-overflow CVEs into compile-time impossibilities. That matters the moment you point it at a file you didn't create.
Architecture: Nine Crates, Four Layers
sheathe is nine crates arranged in four layers, each crate with one responsibility:
Foundation sheathe-core Media model: streams, samples, timing, errors
sheathe-mp4 ISO-BMFF box reader/writer, CMAF fragments
sheathe-crypto CENC encryption (all four schemes + DRM pssh)
sheathe-ts MPEG-2 transport stream demux (PAT/PMT/PES)
sheathe-es Raw elementary stream demux (Annex B, ADTS)
Pipeline sheathe-dash DASH MPD generation
sheathe-hls HLS master + media playlist generation
Application sheathe-cli The `sheathe` binary
sheathe CLI crate (thin wrapper over sheathe-cli)
The dependency graph points strictly down. A manifest crate composes foundation crates but never reaches sideways into another manifest crate. This is where Rust’s crate boundary earns its keep: the MP4 writer cannot accidentally call into the DASH MPD generator, because the compiler won’t let it. Nine shallow crates, each small enough to hold in your head and test in isolation.
The whole workspace is ~6,000 lines of Rust across 41 source files, with 34 tests that verify everything from AES-128-CTR NIST vectors to AVC codec-string derivation — and an integration harness that diffs output against Shaka Packager itself.
The Read Side: Demuxing MP4, MPEG-TS, and Elementary Streams
Before sheathe can write CMAF segments, it has to read the source. Three input paths are wired:
MP4 demux (sheathe-mp4). A dependency-free, cursor-based box reader traverses the ISO-BMFF hierarchy: moov → trak → mdia → minf → stbl. From the sample table (stts/ctts/stsc/stsz/stco/stss) it reconstructs every sample's decode time, composition offset, size, offset, and keyframe flag — the exact information the fragmenter needs to slice the stream into segments. The box reader is a simple state machine over a &[u8] reference; there is no allocation beyond the BoxIterstack, and no unsafe.
// Top-level box traversal: find every box at the current nesting level,
// yielding (type, body, next_offset) for each.
pub fn top_level(data: &[u8]) -> impl Iterator>> {
// ... purely a Cursor over the byte slice
}
MPEG-TS demux (sheathe-ts). Transport streams are a different beast entirely — 188-byte packets carrying program-specific information (PAT/PMT) and packetized elementary stream (PES) payloads. sheathe-ts parses PAT to find the program, PMT to find the streams, PES headers to reassemble access units, and Annex B / ADTS parsers to extract H.264, HEVC, and AAC samples. It synthesizes avcC/hvcC/mp4a sample entries from the elementary stream headers, so the output segment's moov carries the decoder configuration the player needs — all without an external probe tool.
Elementary stream demux (sheathe-es). For raw .h264, .hevc, and .aac files, sheathe detects the format from the extension and content signature, splits the stream into access units (via Annex B start-code scanning for video, ADTS frame sync for audio), and feeds them to the same fragmenter. No container, no problem.
The Write Side: CMAF Segments from a Stream of Samples
Once the demuxer has produced a stream of Sample structs (data, decode timestamp, composition offset, duration, keyframe flag), the fragmenter decides where to cut. The policy is the same one Shaka uses: wait until the accumulated samples exceed the target segment duration (default 6 seconds), then cut at the next keyframe — because a segment that starts on a non-keyframe cannot be decoded independently, and DASH/HLS switching depends on clean boundaries.
pub fn push(&mut self, sample: Sample) -> Result<()> {
let elapsed = sample.dts.saturating_sub(self.current_start);
let may_cut = !self.policy.keyframes_only || sample.is_segment_boundary();
if !self.current.is_empty() && may_cut && elapsed >= self.target_ticks {
self.cut(sample.dts);
}
if self.current.is_empty() {
self.current_start = sample.dts;
}
self.current.push(sample);
Ok(())
}
Each segment is written as an fMP4 file: a styp box (segment type), a sidx box (segment index for seeking), a moof box (movie fragment — the sample table for this segment), and an mdat box (the actual sample data). The init segment carries ftyp + moov + mvex, describing the codec configuration once, at the start.
The box writer is a 75-line nestable builder with automatic size backpatching — no pre-computed table of contents. begin("moof"), write child boxes, end() — and the 32-bit size in the header is patched to the correct value. This is the same pattern Shaka uses; sheathe just does it in 1/10th the code.
The Hard Part: Common Encryption Done Correctly
Encryption is where most packagers get complex. CENC (ISO/IEC 23001–7) defines four protection schemes, and sheathe implements all of them:
SchemeCipherModeVideo PatternAudiocencAES-128-CTRFull-regionNAL-aware subsamplesWhole-samplecensAES-128-CTRPattern (1:9)Crypt 1 block, skip 9Whole-samplecbc1AES-128-CBCFull-regionPer-sample IV, block-alignedWhole-samplecbcsAES-128-CBCPattern (1:9)Crypt 1, skip 9, constant IVWhole-sample
The sheathe-crypto crate is a 457-line module built on the RustCrypto aes crate. The AES-128-CTR path runs a continuous keystream across protected byte ranges, skipping clear bytes and pattern-skipped blocks without advancing the counter. The AES-128-CBC path chains ciphertext blocks, resetting to the constant IV at each subsample boundary under pattern encryption. Both are validated against NIST SP 800-38A test vectors, and against Shaka Packager's output on real media.
Every scheme is ffmpeg decrypt+decode verified. That is: sheathe encrypts a sample, produces an fMP4 segment, and then ffmpeg -decryption_key ... -i segment.mp4 -f framemd5 confirms every video frame and every audio frame decodes to the same MD5 as the clear original. No "probably encrypted" — provably correct.
Multi-DRM pssh
Encrypting the bytes is half the job; the other half is telling the player’s DRM module how to decrypt them. Each DRM system gets a pssh box in the init segment, generated directly from the raw key (no key server required):
Common — a version-1 box listing the KID, used by W3C clear-key and urn:mpeg:dash:mp4protection.
Widevine — a version-0 box carrying a WidevinePsshData protobuf with the KID and protection-scheme fourcc.
PlayReady — a version-0 box wrapping a PlayReady Object with a WRMHEADER 4.0.0.0 (KID, key length, ALGID, and an AES-ECB checksum proving key possession).
Each box byte-matches Shaka Packager’s --protection_systems output. Not approximately — identically.
Key Rotation
For long content, rotating keys reduces the exposure window of any single key. sheathe supports --crypto-period-duration : each period derives a new content key by left-rotating the base key (Shaka's naive raw-key scheme), signalled per segment via seig sample groups (sbgp/sgpd), a zero-KID init tenc, and a per-period pssh in each moof. Every segment decrypts to the clear baseline under its derived key.
Manifest Generation: DASH and HLS
With segments written, the packager generates the manifests that players consume.
DASH. The MPD is an XML document in the static (type="static"), on-demand profile, using SegmentTemplate + SegmentTimeline — the same profile Shaka produces for VOD. Each representation gets a element with its codec string (RFC 6381), resolution, bandwidth, and a that lists each segment's exact duration. Equal-duration runs are collapsed with r= (repeat count), so a 2-hour movie doesn't produce 1,200 ~~elements.~~
The codec strings themselves are derived from raw box data — avcC → avc1.640028, hvcC → hvc1.1.6.L93.90, av1C → av01.0.04M.08, esds → mp4a.40.2 — in a pure-Rust, zero-dependency codec-string module with tests pinned against known outputs.
// avc1.PPCCLL from avcC
fn avc_string(fourcc: &[u8; 4], avcc: &[u8]) -> Option {
let (profile, compat, level) = (*avcc.get(1)?, *avcc.get(2)?, *avcc.get(3)?);
let prefix = std::str::from_utf8(fourcc).ok()?;
Some(format!("{prefix}.{profile:02x}{compat:02x}{level:02x}"))
}
HLS. The master playlist lists video variants as #EXT-X-STREAM-INF entries with combined bandwidth and codecs (video + audio folded together, per the spec), bound to an #EXT-X-MEDIA audio rendition group. Media playlists carry #EXTINF per-segment durations, #EXT-X-MAP pointing to the init segment, and #EXT-X-KEY encryption signalling (SAMPLE-AES for cbcs, SAMPLE-AES-CTR for cenc) — the full HLS encrypted-fMP4 contract.
~~The Oracle: Differential Testing Against Shaka Packager~~
sheathe adopts the “revelo method”: implement in pure Rust, then differential-test against a reference oracle. For a packager, the oracle is Shaka Packager running the same input file through the same output configuration.
~~The oracle harness (just oracle input.mp4) runs both packagers on the same source, then diffs:~~
Segment counts. Do sheathe and Shaka produce the same number of media segments?
Canonical MPD. XML canonicalization (xmllint --c14n) strips formatting to compare structure — same representations, same timelines, same codec strings.
ffmpeg decode. Both outputs are fed through ffmpeg -f framemd5 to confirm the video renders identically.
This is the same methodology that keeps revelo’s 180+ parsers byte-identical to MediaInfoLib and viser’s convex hulls correct to the numerical method. Nothing ships until its bytes are verified against the oracle. If Shaka and sheathe disagree, the bug is in sheathe until proven otherwise.
~~What Makes This Genuinely Hard~~
~~Rust makes it safe; the domain makes it hard. The genuinely difficult parts of building a packager are worth naming:~~
NAL-aware subsample encryption is format-specific. AVC and HEVC samples cannot be encrypted blindly — the NAL unit headers and slice headers must stay clear for a player to parse the stream. The senc box carries a table of (clear_bytes, encrypted_bytes) per sample, and getting the boundary exactly right requires knowing the NAL unit length size from the avcC/hvcC configuration. Get it wrong, and the player produces a green screen or a crash.
The full CENC scheme matrix has subtle interactions. cbcs uses a constant IV across all samples; the other three derive per-sample IVs. cbc1 chains CBC across subsamples; cbcs resets to the constant IV at each subsample. Pattern encryption applies to video only — audio under cens/cbcs is whole-sample encrypted with Pattern::NONE. DASH-IF conformance requires saiz/saio auxiliary-information boxes with the saio offset backpatched to point at the senc data. Every one of these interactions earned its own differential test.
PlayReady pssh is deeply non-obvious. The PlayReady Object wraps a WRMHEADER with a KID that uses swapped GUID byte order (the first three fields of the UUID are written little-endian, not network order), and an AES-ECB checksum of the object. Building this from the raw key requires implementing a subset of the PlayReady binary format that no RFC describes — the oracle is the only spec.
The box model rewards getting it right the first time. ISO-BMFF boxes nest arbitrarily: moov contains trak, which contains mdia, which contains minf, which contains stbl, which contains stsd, which contains the actual sample entry with its decoder configuration. A missing box, a wrong offset, or an incorrect size computation produces a file that looks valid to a file browser and silently fails to play. The reader and writer together form a round-trip that is verified on every test asset.
~~The Result~~
sheathe is a pure-Rust HLS/DASH/CMAF media packager at ~6,000 lines, validated against Shaka Packager as the reference oracle. It demuxes MP4, MPEG-TS, and raw elementary streams; writes CMAF init and media segments with correct box structure and codec strings; applies all four CENC encryption schemes with multi-DRM pssh (Widevine, PlayReady, Common) and key rotation; and emits DASH MPDs and HLS playlists that match Shaka’s output.
# Package an MP4 into 6s CMAF segments with both DASH and HLS manifests.
sheathe package input.mp4 --out site/ --segment-duration 6 --dash --hls
# Encrypt with CENC cbcs (AES-CBC pattern) and Widevine+PlayReady DRM.
sheathe package input.mp4 --out site/ --dash --hls \
--enc-scheme cbcs \
--enc-key abcd1234...:deadbeef... \
--protection-systems common,widevine,playready
# Probe an MPEG-TS file — no ffprobe required.
sheathe probe input.ts
~~cargo install sheathe~~
sheathe doesn’t yet do live streaming, trick-play, or WebM input. Those are tracked on the roadmap, each gated behind an oracle validation checkpoint. What ships today is the complete VOD pipeline — the subset that covers ~90% of production packaging work — built in unsafe-free Rust, with no C dependencies, and every output differential-tested against the best open-source packager available.
~~One More Thing: The Pure-Rust Probe~~
A packager needs to know what it’s packaging. Resolution, frame rate, codec, bitrate — the numbers that determine how segments get sized and manifests get filled. The default way to get those numbers is to shell out to ffprobe.
sheathe doesn’t need to. Its MP4 demuxer reads sample-table metadata directly from the box hierarchy; its MPEG-TS demuxer reconstructs stream descriptions from PAT/PMT/PES headers; its elementary-stream path synthesizes sample entries from the Annex B/ADTS byte stream itself. The sheathe probe command returns the same information ffprobe does, with no external binary on the path.
For the full metadata picture — the kind that covers 177+ media formats with byte-identical output to MediaInfoLib — there’s revelo, another pure-Rust project that integrates as a drop-in probe engine. But for the subset a packager actually needs, sheathe is self-contained: the tool that segments video is also the tool that reads it. One binary, one dependency chain, one cargo install.
~~What I’d Want You to Take From This~~
~~If there’s a lesson worth carrying out of this project, it’s simpler than the topic implies:~~
Choose an oracle you can diff against. Shaka Packager is open source, battle-tested on billions of streams, and emits deterministic output. Every feature sheathe ships was validated by running both tools on the same input and comparing the result. Byte-identical output is the only claim worth making.
The crate boundary is a design tool. Nine crates, each with exactly one concern, separated by the compiler. The MP4 writer cannot reach into the DASH generator; the crypto crate knows nothing about the container format. Wide and flat, not deep and tangled.
The hardest bugs are interactions, not algorithms. AES-CTR is 40 lines. Getting the saiz offset to backpatch correctly after the senc is written — that's where the hours go. The differential harness catches these; no amount of code review does.
Packaging is infrastructure, not magic. A packager is a program that reads video, cuts it into pieces, and writes a description of the pieces. Doing it correctly is a lot of work; doing it in pure Rust is less work than you’d think, and the result is safer, simpler, and faster to install than the C++ alternative.
~~sheathe is open source at github.com/vbasky/sheathe, under MIT OR Apache-2.0.~~
The author is also building revelo (a pure-Rust MediaInfoLib port) and viser (a content-adaptive video encoding optimizer). The three projects form a suite: probe → ladder → package — the Delivery lane of a pure-Rust media ecosystem, end to end.

Catharsis for Noisy Audio: A Pure-Rust Restoration Toolkit with No ffmpeg and No Black Boxes

Vikram Bhaskaran — Thu, 18 Jun 2026 08:06:11 GMT

How I built cathar — denoise, de-hum, de-click, de-clip, de-reverb, de-ess, voice isolation, bandwidth extension and loudness, every stage a classic inspectable algorithm, in a single static Rust binary.
A language model helped draft this article
Every recording arrives a little broken. A camera mic picks up air-conditioning hiss. A guitar DI hums at 50 Hz. A lapel mic clips on a loud laugh. A click track bleeds into a take. A Zoom call reverberates off bare walls. A phone recording has no top end above 8 kHz.
The usual fix is a pile of tools you don’t control. You shell out to ffmpeg, route through a chain of VST plugins you can’t script, or hand the whole thing to a neural denoiser that either fixes it or smears it into a watery, artifacted mess — and you can’t tell which until you’ve already listened to the output.
I wanted the opposite: a toolkit where every stage is a named, documented algorithm I can read end to end; a pipeline that never quantises between steps; and a binary with no system dependencies whatsoever — no ffmpeg, no libsndfile, no pkg-config. That's cathar.
The name is from Greek katharós (καθαρός) — “pure, clean” — the same root as catharsis, a cleansing. That’s the whole job: take a noisy recording and give back clean audio.
What it is
Cathar is an audio toolkit for any recording, written in pure Rust. It works on a standalone audio file (WAV, MP3, FLAC, OGG, M4A) just as readily as the audio track inside a video (MP4, MKV) — video is never required, the audio is pulled straight out of the container. It does three classes of work and writes a clean 32-bit float WAV:
Restore — denoise, de-hum, de-click, de-clip, de-reverb.
Enhance — de-ess, breath removal, voice isolation, bandwidth extension.
Level — loudness (LUFS) and peak normalisation for delivery.
cathar denoise interview.mp4 --out clean.wav # noisy interview off a camera -> clean dialogue
cathar noiseprint room_tone.wav --out room.np.json # learn the room tone from a silent clip
cathar denoise interview.mp4 --noiseprint room.np.json --out clean.wav # then denoise with it
cathar dehum recording.wav --freq 60 # kill 60 Hz mains buzz
cathar declick recording.wav # interpolate impulse clicks
cathar declip recording.wav # rebuild clipped peaks
cathar normalize recording.wav --target -16 # to -16 LUFS (podcast)
Decoding is symphonia, the FFT is realfft/rustfft, WAV writing is hound. A single cargo build gives you a self-contained binary that runs anywhere — a container, a CI job, a Lambda — with no install step.
The design stance
Five principles drive every decision, and they’re worth stating up front because they explain why the rest of the article looks the way it does:
Pure Rust — no ffmpeg, no C/C++ FFI, no pkg-config; one cargo build produces a self-contained binary.
Lossless float pipeline — decode → f32 → process → 32-bit float WAV; nothing is quantised between stages.
Composable — every effect is a plain fn(&[f32], …) -> Vec; chain them in any order, in the CLI or as a library.
Inspectable DSP — classic, documented algorithms (STFT subtraction, Wiener, IIR notches, cubic interpolation), not opaque models.
Deterministic — deterministic per-file output; batch processes files concurrently.
The one I care most about is inspectable. When a denoiser leaves a metallic “musical noise” artifact, I want to know it came from too-aggressive spectral subtraction and turn a specific knob — not re-roll a model and pray. Every algorithm below is one you can look up, reason about, and predict.
The backbone: an STFT loop
Cathar decodes to interleaved f32 PCM — one Vec per channel, at the file's native sample rate — and most reduction stages run as the same loop: a short-time Fourier transform → modify the spectrum → inverse STFT. The denoiser uses a 2048-point FFT with a 512-sample hop (75% overlap) and a Hann window on both analysis and synthesis, reconstructed by overlap-add:
The 75% overlap and the Hann-window-on-both-ends is what makes the reconstruction seamless: with constant overlap-add the windows sum to unity, so untouched frames come back bit-for-bit and modified frames don’t click at their boundaries. Phase is never altered — only the magnitude spectrum is reshaped — which is exactly what keeps processed speech sounding natural rather than phasey and underwater.
Everything that follows is a variation on “what do we do to the magnitudes.”
Reducing noise: reshaping the spectrum
Denoise is the headline. Two flavours share the frame loop above:
Spectral subtraction (default) estimates the noise magnitude in each frequency bin and subtracts α × it, held above a spectral floor β·mag. That floor is the artifact knob: subtract too hard and you carve isolated spectral peaks out of the noise floor that ring as "musical noise"; the β·mag floor leaves a thin, natural-sounding bed instead. α from 1→6 takes you gentle→aggressive.
Wiener filter (--wiener) instead applies the statistically optimal per-bin gain S / (S + N) derived from the estimated signal and noise power. It's smoother on stationary, broadband noise like tape hiss or fan rumble, because the gain rolls off continuously rather than thresholding.
Both need a noise estimate, and that comes from one of two places. Minimum-statistics assumes the quietest ~15% of frames are essentially noise and builds the profile from them — zero configuration, works on anything. But if you can spare two seconds of true room tone, noiseprint does better: it captures the exact per-bin magnitude spectrum of a dedicated silent segment, serialises it to JSON, and feeds it back as a precise subtraction target. Learn it once, reuse it across every take from the same session.
De-hum is a different problem and gets a different tool. Mains hum is not broadband — it’s a fundamental at 50 or 60 Hz plus a stack of harmonics. Spectral subtraction would be the wrong hammer; instead dehum builds a cascade of second-order IIR notch biquads (Q = 30) tuned to the base frequency and each harmonic up to Nyquist. A high-Q notch removes a razor-thin slice at each hum frequency and leaves everything around it untouched, so the voice keeps its body while the buzz disappears.
De-reverb tackles the smear of a live room. It runs a two-pass spectral-decay gate: it tracks each frequency bin’s envelope with an 8 ms attack and a 50 ms release, then gates bins that are sitting near their own reverb-decay floor — i.e. energy that is decaying rather than being freshly driven. Direct sound passes; the ringing tail gets pulled down.
Voice isolation keeps speech and gates everything else. An energy-based voice-activity detector runs on 20 ms frames, with gap-filling (bridge silences under 120 ms so words don’t get chopped) and minimum-segment dropping (discard blips under 50 ms that are clearly not speech). Non-speech regions are then spectrally gated — and if you hand it a noiseprint, the gate tightens around the actual noise spectrum.
De-ess chases sibilance — the harsh “sss” and “tss” that lavs and condensers exaggerate. It runs a tighter STFT (2048/256) and, wherever the high-frequency power ratio above a crossover frequency exceeds a threshold, applies frequency-dependent compression to just that band. Because it keys on the ratio, it only acts on actual sibilant frames and leaves the rest of the vocal alone.
Breath removal is subtle on purpose. It flags the frames immediately before a speech onset (≤ 150 ms), high-passes them at 200 Hz, and mixes the result back 40/60 dry/wet — so breaths are softened, not surgically deleted, which is what keeps the delivery sounding human instead of gated and robotic.
Repairing damage: reconstructing samples
Noise reduction reshapes a spectrum. The repair tools do something more aggressive — they decide some samples are wrong and rebuild them.
De-click targets impulse damage: vinyl ticks, digital dropouts, mouth clicks. It runs a sliding window, computes the local RMS, and flags any sample that spikes past threshold × RMS as a click. Flagged runs are then replaced by cubic-Hermite interpolation across the gap — fitting a smooth curve through the good samples on either side so the repair is inaudible rather than a hole.
De-clip handles the opposite failure: a signal recorded too hot, with its peaks flattened against the ceiling. It detects runs sitting at or above the clip threshold, extends the shoulders ±4 samples to catch the full flattened region, and reconstructs the missing peak with the same cubic-Hermite interpolation — inferring the curve the waveform would have taken if the converter hadn't run out of headroom.
Both are honest reconstructions of small, localised damage — not magic — and because they only touch flagged samples, undamaged audio passes through untouched.
Enhancing and levelling
Enhance is bandwidth extension for dull, band-limited sources — old phone recordings, low-bitrate exports. It windowed-sinc resamples up to the target rate, then uses spectral band replication: it folds the existing top band of the signal up into the empty highs with a tiled rolloff, synthesising plausible high-frequency detail where the source simply had none. It’s the only stage that resamples; everything else runs at the source rate.
Normalize is delivery. Peak mode scales so the loudest sample hits a dBFS target; loudness mode scales by RMS to approximate a LUFS target. The rough targets: -23 for broadcast (EBU R128), -16 for podcast, -14 for streaming.
Run them however you like
Every command reads any supported format and writes a 32-bit float WAV, so you can run them in any order, chain them, or batch a directory:
cathar batch --indir ./raw --outdir ./clean --dehum 60 --normalize -16
And because each stage is independent and frame-synchronous, the same input always produces the same output — no seeds, no nondeterminism, which matters the moment you put this in a reproducible pipeline.
It’s a library first, a CLI second
The CLI is a thin clap wrapper over the cathar crate. Because every effect is a plain function over &[f32], the same engine drops straight into a Rust program:
use cathar::{AudioData, SpectralDenoiser, Denoiser, dehum, normalize_loudness};
let audio = AudioData::from_file("interview.mp4")?; // symphonia decode → f32
let sr = audio.sample_rate;
let clean = SpectralDenoiser::default()
.denoise(&audio)?
.map_channels(|ch| dehum(ch, sr, 60.0, 5))
.map_channels(|ch| normalize_loudness(ch, -16.0));
clean.to_file("clean.wav")?; // 32-bit float WAV via hound
AudioData is just { sample_rate, channels: Vec> }. map_channels applies any &[f32] -> Vec effect to every channel independently. Learn a noise print once and reuse it for a tighter subtraction:
use cathar::{AudioData, SpectralDenoiser, Denoiser, learn_noise_print};
let print = learn_noise_print(&AudioData::from_file("room_tone.wav")?)?;
let audio = AudioData::from_file("interview.mp4")?;
let clean = SpectralDenoiser::with_noise_print(print, /* alpha */ 3.0, /* beta */ 0.01)
.denoise(&audio)?;
clean.to_file("clean.wav")?;
The whole public surface is a handful of free functions — dehum, declick, declip, dereverb, voice_isolate, deesser, breath_remove, bandwidth_extend, normalize_peak, normalize_loudness, generate_wave — plus the Denoiser trait, SpectralDenoiser, and NoisePrint. Nothing hidden, nothing you can't call directly.
The lossless float pipeline
One detail that’s easy to miss but matters: cathar decodes to 32-bit float, processes in float, and writes 32-bit float WAV. Nothing is quantised back to 16-bit between stages. When you chain denoise → dehum → declip → normalize, each stage hands the next full-precision floats, so quantisation error never accumulates across a five-stage chain. The cost is bigger intermediate files; the payoff is that a long restoration chain doesn't slowly add its own noise floor.
The architecture
A deliberately small two-crate workspace — a library and the binary that drives it:
cathar/
├─ crates/
│ ├─ cathar/ # the engine: decode (symphonia) · DSP · encode (hound)
│ └─ cathar-cli/ # the `cathar` binary — clap subcommands over the engine
└─ docs/ # banner + assets
symphonia (built with features = ["all"]) decodes every supported container and codec to f32; realfft/rustfft sit behind every STFT stage; hound writes the float WAV; serde/serde_json serialise the NoisePrint. The optional ml feature wires in candle as scaffolding for a future learned denoiser.
Honest about the edges
Cathar is 0.1.x, and I'd rather tell you what it doesn't do yet than oversell:
Loudness is an RMS-based LUFS approximation, not true K-weighted gated EBU R128. Good enough to land a podcast in the ballpark; not a broadcast-compliance tool.
The learned denoiser isn’t built. The ml feature wires in candle, but the neural model is scaffolding. The classic DSP chain is what's implemented and unit-tested.
Batch fans out across the rayon thread pool — files process in parallel, and a bad file is reported and skipped rather than aborting the run.
Only enhance resamples; mixed-rate inputs aren't yet handled uniformly across every stage.
Try it
cargo install --path crates/cathar-cli
cathar wave --out test.wav --duration 3 --freq 440 --noise 0.15 # make a noisy tone
cathar denoise test.wav --out clean.wav # clean it
The code is on GitHub: github.com/vbasky/cathar. If you’ve ever wanted a denoiser whose source you can actually read — and predict — this one’s for you.

Somnia — A Type-Safe ORM for SurrealDB

Vikram Bhaskaran — Sun, 14 Jun 2026 15:10:18 GMT

Stop Hand-Writing SurrealQL Strings in Rust
Every Rust developer who has worked with SurrealDB knows the sinking feeling. You write a query. It compiles. It runs. And then, at 3 AM, production throws a parse error because a record ID got escaped wrong, or a column name had a typo, or a UUID got treated as an arithmetic expression inside a string interpolation.
You’re not alone. I built Somnia because I was tired of debugging query strings at runtime when Rust’s type system was sitting right there, ready to help.
The Stringly-Typed Nightmare
Let me paint a picture. Here’s what a typical SurrealDB query looks like in a Rust codebase using the raw surrealdb crate:
let query = format!(
"SELECT id, title, body, published_at FROM post \
WHERE title = '{}' AND published_at < d'{}' \
ORDER BY published_at DESC \
LIMIT {}",
title, cutoff, limit
);
let result: Vec = db.query(query).await?.take(0)?;
What could possibly go wrong? Let me count the ways:
That ‘title’ field in the string — rename title to headline in your struct, and the compiler won’t catch that the query still references the old column name.
That d’{}’ datetime literal — forget the d prefix and SurrealDB silently interprets your timestamp as a string. Type mismatch bugs that surface days later.
That unescaped title value — what if the title contains a single quote? Oops.
The id field — SurrealDB record IDs have subtle parsing rules. A UUID with dashes gets interpreted as type::thing — 550e8400 — e29b — 41d4 — a716–446655440000 (an arithmetic expression) unless wrapped in backticks.
The take(0) — hope you remembered the right index. Deserialization targets a positional result set; mismatch it and you’re deserializing the wrong thing.
Every one of these is a runtime error. Some are silent. Some crash. None are caught by the compiler. And when you’re maintaining dozens of queries across a growing codebase, the cognitive load adds up fast.
Enter Somnia: Schema as Code, Queries as Types
Somnia flips the model. Instead of writing queries and hoping they match your types, you define your types and generate the queries. The Rust struct becomes the single source of truth for both your application model and your database schema.
use somnia::SurrealRecord;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Serialize, Deserialize, SurrealRecord)]
#[table("post")]
struct Post {
#[field(thing)]
id: Thing,
title: String,
body: String,
published_at: Option,
}
That #[derive(SurrealRecord)] annotation does heavy lifting. It generates:
Typed column accessors — Post::title() returns a Column
A table() method to enter the query builder
The SurrealSchema trait for DDL generation (more on that later)
Post::all() — a column set for SELECT *
Now queries look like this:
use somnia::query::ident;
let query = Post::table()
.select(Post::all())
.filter(Post::title().eq("Hello, world"))
.filter(ident("published_at").less_than("2025-01-01"))
.order_desc(ident("published_at"))
.limit(10)
.to_surrealql();
Run this and to_surrealql() produces:
SELECT * FROM post
WHERE title = 'Hello, world' AND published_at < '2025-01-01'
ORDER BY published_at DESC LIMIT 10
Notice what happened behind the scenes:
Post::title() is a typed accessor. If you rename the field in your struct, the code won’t compile.
The table name comes from the #[table(“post”)] attribute — define it once, never mistype it again.
Values are automatically escaped with the right SurrealQL literal syntax (strings in single quotes, datetimes with the d prefix, UUIDs with the u prefix).
The entire query is a builder chain. The compiler type-checks every expression, column reference, and operator. You can’t accidentally use a string column in an arithmetic comparison. You can’t filter on a field that doesn’t exist.
Beyond SELECT: Mutations That Make Sense
Somnia handles the full CRUD lifecycle with the same type-safe builder pattern.
CREATE
let query = Post::table()
.create()
.record("post:hello-world")
.set_lit("title", "Hello, world")
.set_lit("body", "This is a post about SurrealDB.")
.set_lit("published_at", "2025-06-15")
.returning(Returning::After)
.to_surrealql();
Generates:
CREATE post:hello-world SET
title = 'Hello, world',
body = 'This is a post about SurrealDB.',
published_at = '2025-06-15'
RETURN AFTER
Returning::After returns the newly created record. Use Returning::Before for the pre-update snapshot, Returning::Diff to see only changed fields, or Returning::None to suppress the return value.
You can also use the CONTENT pattern if you already have a fully-formed struct:
Post::table()
.create()
.content(post_obj) // post_obj: &Post
.returning(Returning::After)
.to_surrealql()
UPDATE with SET, MERGE, and CONTENT
Somnia supports all three SurrealDB update strategies:
// SET — assign specific fields
Post::table()
.update("post:hello-world")
.set_lit("title", "Updated title")
.set_lit("body", "New body content")
.to_surrealql();
// MERGE - deep-merge a JSON object
Post::table()
.update("post:hello-world")
.merge(json!({"title": "Merged title"}))
.to_surrealql();

// CONTENT - replace the entire record
Post::table()
.update("post:hello-world")
.content(&updated_post)
.to_surrealql();
UPSERT
New in 0.4.1, UPSERT gives you insert-or-update with a single call:
Post::table()
.upsert("post:maybe-exists")
.set_lit("title", "Exists or not, here it is")
.returning(Returning::After)
.to_surrealql();
DELETE with Safety
Post::table()
.delete("post:obsolete")
.returning(Returning::Before) // get the record before removing it
.to_surrealql();
The “Mutate Then Reselect” Pattern
A common workflow: you create a record, and you need the fully-hydrated version (with computed fields, timestamps, defaults) immediately. Somnia’s then_select() chains a mutation with a follow-up SELECT in a single round-trip:
let query = Post::table()
.create()
.content(&new_post)
.returning(Returning::After)
.then_select(
Post::table()
.project(vec![
Post::title(),
Post::published_at(),
])
.limit(1)
);
The generated query joins two statements with ; — SurrealDB executes them as a unit and returns both result sets.
Complex Filtering: ANDs, ORs, and Everything in Between
Real queries rarely stop at a single WHERE clause. Somnia’s expression tree composes naturally:
use somnia::expr::Expr;
let is_published = ident("published_at").less_than_or_equal("2025-06-15");
let is_featured = Post::title().contains("featured");
let author_is_vikram = Post::title().eq("Vikram");
let query = Post::table()
.select(Post::all())
.filter(is_published.and(is_featured.or(author_is_vikram)))
.limit(50)
.to_surrealql();
Generates:
SELECT * FROM post
WHERE (published_at <= '2025-06-15' AND (title CONTAINS 'featured' OR title = 'Vikram'))
LIMIT 50
Parentheses are automatically inserted for correct precedence. The expression tree handles arbitrary nesting — chain as many .and() and .or() combinators as you need.
You can also compare columns against columns, not just literals:
use somnia::expr::Expr as _; // for .gt(), .lt(), .eq()
let query = Post::table()
.select(Post::all())
.filter(ident("updated_at").greater_than(ident("created_at")))
.to_surrealql();
Schema as Code: Your Struct IS Your DDL
This is where Somnia truly shines. Every SurrealRecord automatically generates schema DDL. Call Post::up() and you get:
DEFINE TABLE IF NOT EXISTS post SCHEMAFULL PERMISSIONS FULL;
DEFINE FIELD IF NOT EXISTS title ON TABLE post TYPE string;
DEFINE FIELD IF NOT EXISTS body ON TABLE post TYPE string;
DEFINE FIELD IF NOT EXISTS published_at ON TABLE post TYPE option;
The Post::id field — marked with #[field(thing)] — is skipped from the generated DDL because SurrealDB handles the id field automatically.
Call Post::down() and you get:
REMOVE TABLE IF EXISTS post;
You can customize the table definition with attributes:
#[derive(SurrealRecord)]
#[table("post", permissions = "NONE")]
struct Post { ... }
// DEFINE TABLE IF NOT EXISTS post SCHEMAFULL PERMISSIONS NONE;
#[derive(SurrealRecord)]
#[table("event_log", schemaless)]
struct EventLog { ... }
// DEFINE TABLE IF NOT EXISTS event_log SCHEMALESS PERMISSIONS FULL;
This is powerful because it eliminates the drift between your application types and your database schema. When a field is added, renamed, or retyped, the generated DDL changes automatically. Your CI pipeline can compare the generated DDL against the production schema to surface discrepancies before they ship.
Migrations: Diesel-Style, SurrealDB-Ready
If you’ve used Diesel, Somnia’s migration system will feel immediately familiar. Migrations are timestamped folders containing up.surql and down.surql files:
migrations/
├── 2025-06-15-142300_create_posts/
│ ├── up.surql
│ └── down.surql
├── 2025-06-15-142500_seed_categories/
│ ├── up.surql
│ └── down.surql
└── 2025-06-15-142700_add_tags/
├── up.surql
└── down.surql
Each up.surql might look like:
-- up.surql
DEFINE TABLE IF NOT EXISTS tag SCHEMAFULL PERMISSIONS FULL;
DEFINE FIELD IF NOT EXISTS name ON TABLE tag TYPE string;
DEFINE INDEX IF NOT EXISTS idx_tag_name ON TABLE tag COLUMNS name UNIQUE;
And the corresponding down.surql:
-- down.surql
REMOVE TABLE IF EXISTS tag;
Migrations are applied in timestamp order, tracked in a special _somnia_migrations table (auto-created on first run). Use them from code:
let client = SomniaClient::connect(
"ws://localhost:8000",
"root", "root",
"myns", "mydb"
).await?;
let migrator = client.migrator("migrations");
// Apply all pending migrations
migrator.run().await?;
// Check status
for m in migrator.status().await? {
let icon = if m.applied { "✓" } else { " " };
println!("{} {}", icon, m.name);
}
// Revert the most recent migration
migrator.revert_last().await?;
// Revert everything
migrator.revert_all().await?;
Or use the CLI:
somnia migration generate add_tags # generate a new migration scaffold
somnia migration run # apply pending migrations
somnia migration revert # revert the most recent
somnia migration redo # revert and re-apply the latest (useful during development)
somnia migration list # see what's applied and what's pending
The CLI connects via flags or environment variables:
somnia migration run \
--endpoint ws://localhost:8000 \
--user root --pass root \
--ns myns --db mydb
Graph Relations: Typed Edges in SurrealDB
SurrealDB’s graph model is one of its killer features. Somnia supports typed RELATE queries:
// Define edge and node types
#[derive(SurrealRecord)]
#[table("authored")]
struct Authored {
#[field(thing)]
id: Thing,
since: String,
}
#[derive(SurrealRecord)]
#[table("author")]
struct Author {
#[field(thing)]
id: Thing,
name: String,
}
// Relate with a SET body
let relate = Authored::table()
.relate(Author::table(), "author:vikram", Post::table(), "post:hello-world")
.set_lit("since", "2025-01-01")
.to_surrealql();
Generates:
RELATE author:vikram -> authored -> post:hello-world SET since = '2025-01-01'
The type parameters track which node types the edge connects. The compiler won’t let you accidentally relate a Post to a Comment through an Authored edge if those types don’t match the edge’s intended schema.
Geometry Types: Latitude, Longitude, and Type Safety
If you’re building location-aware applications, SurrealDB’s geometry support is important. Somnia v0.3.0 added first-class geometry types with proper GeoJSON serialization:
use somnia::types::{Point, LineString, Polygon};
#[derive(SurrealRecord)]
#[table("venue")]
struct Venue {
#[field(thing)]
id: Thing,
name: String,
location: Point,
service_area: Polygon,
}
// The coordinates serialize as proper GeoJSON
let venue = Venue {
id: Thing::new("venue", "central-park"),
name: "Central Park".into(),
location: Point::new(-73.9654, 40.7829),
service_area: Polygon::new(vec![
(-74.0, 40.7),
(-73.9, 40.7),
(-73.9, 40.8),
(-74.0, 40.8),
(-74.0, 40.7),
]),
};
When stored in SurrealDB, these serialize as {“type”: “Point”, “coordinates”: [-73.9654, 40.7829]} — the format SurrealDB expects for its geometry type. Without this, SurrealDB would store coordinates as a plain array, losing the ability to use geospatial indexes and queries.
Before v0.3.0, geometry was stored as bare arrays, which silently broke SurrealDB’s geospatial features. The fix was a breaking change, but it was necessary — correctness over convenience.
Record ID Escaping: The Bug You Didn’t Know You Had
SurrealDB’s record ID parser has non-obvious rules. Consider these keys:
hello — simple, works fine as post:hello
U+ID — contains +, which is an operator, so it needs escaping
550e8400-e29b-41d4-a716–446655440000 — a UUID without backticks gets parsed as type::thing — 550e8400 — e29b — …, an arithmetic expression
Somnia v0.3.0 added proper key escaping. The Key enum auto-detects the escaping strategy:
Bare integers render as post:42 (no quotes needed)
Simple ASCII identifiers render as post:hello (no backticks needed)
UUIDs and strings with special characters render as post:⟨550e8400-e29b-41d4-a716–446655440000⟩ (backtick-wrapped)
(Replace ⟨⟩ with actual backticks above — Medium’s rendering can be tricky with backticks inside code blocks.)
This is the kind of detail that silently corrupts data at scale. You might not notice it in development with a handful of records. But when your UUID-based key scheme hits production and queries start returning zero results because the IDs were parsed as arithmetic expressions? That’s a bad day.
Somnia handles it transparently.
The Expression Tree: How It All Works Under the Hood
The query builder isn’t just string concatenation. Somnia constructs an intermediate expression tree that models SurrealQL’s grammar as Rust types.
Every expression node implements two traits:
/// Type-erased rendering. Every node knows how to write itself to a buffer.
trait DynExpr {
fn render_dyn(&self, buf: &mut String);
}
/// Typed expression with a SurrealQL type hint.
trait Expr: DynExpr {
fn ty_hint() -> &'static str; // e.g., "string", "int", "datetime"
}
The expression node types form a tree:
Literal — a typed literal value (string, int, float, bool, datetime, uuid, geometry, JSON)
Column — a typed column reference (generated by Post::title())
Ident — a raw SurrealQL identifier (for expressions the type system doesn’t yet model)
Func — a function call like count(), array::len(), type::thing()
RecordLink — a typed pointer to another record (Thing references)
Raw — an escape hatch for arbitrary SurrealQL fragments
Binary operators — EqExpr, GtExpr, OrExpr, AndExpr, ContainsExpr, NotExpr
Grouped — parenthesized sub-expressions (for precedence)
When you write .filter(Post::title().eq(“Hello”)), you’re constructing an EqExpr node wrapping a Column and a Literal. When you call .and(…), you’re wrapping that in an AndExpr. When to_surrealql() is called, the tree recursively renders each node into the output buffer.
This architecture has two big advantages:
Composability. You can build filter expressions independently, store them in variables, combine them, and reuse them across queries. The Rust borrow checker ensures lifetimes are correct.
Extensibility. Adding a new operator or expression type means implementing DynExpr (and optionally Expr) for a new struct. The rendering pipeline handles the rest.
Connection and Execution: The SomniaClient
Somnia wraps the raw Surreal connection from the surrealdb crate with a thin, typed client:
use somnia::SomniaClient;
#[tokio::main]
async fn main() -> somnia::Result<()> {
let client = SomniaClient::connect(
"ws://localhost:8000", // endpoint
"root", // username
"root", // password
"mynamespace", // namespace
"mydatabase", // database
).await?;
// Typed query - deserializes into Vec
let posts: Vec = client
.query(
&Post::table()
.select(Post::all())
.filter(Post::title().contains("SurrealDB"))
.limit(10)
)
.await?;
// Typed insert - uses $data bind parameters internally
let new_post = Post {
id: Thing::new("post", "hello-world"),
title: "Hello".into(),
body: "World".into(),
published_at: Some("2025-06-15".into()),
};
client.insert(&Post::table().insert(vec![&new_post])).await?;
// Raw escape hatch - returns Vec
let raw: Vec = client
.raw("SELECT count() FROM post GROUP ALL")
.await?;
Ok(())
}
The query() method renders the builder to a string, executes it against SurrealDB, and deserializes the first result set into Vec. The insert() method uses SurrealDB’s bind-parameter feature ($data) for safe record insertion.
The raw() method is your escape hatch. When you need to run a query that Somnia’s builder doesn’t yet support (window functions, advanced graph traversals, or anything niche), raw() gives you a Vec with no ceremony.
The Derive Macro: Anatomy of a SurrealRecord
The #[derive(SurrealRecord)] proc macro is the heart of Somnia’s developer experience. Here’s what it really generates for a struct:
#[derive(SurrealRecord)]
#[table("post")]
struct Post {
#[field(thing)]
id: Thing,
title: String,
published_at: Option,
}
This expands (conceptually) to:
impl SurrealRecord for Post {
fn table_name(&self) -> &'static str { "post" }
fn primary_key(&self) -> Column> { Self::id() }
fn table() -> Table { Table::new("post") }
}
impl Post {
pub fn id() -> Column> {
Column::new("post", "id")
}
pub fn title() -> Column {
Column::new("post", "title")
}
pub fn published_at() -> Column> {
Column::new("post", "published_at")
}
pub fn all() -> ColumnSet {
ColumnSet::new(vec![
Self::id().erase(),
Self::title().erase(),
Self::published_at().erase(),
])
}
}
impl SurrealSchema for Post {
fn up() -> String {
// DEFINE TABLE IF NOT EXISTS post SCHEMAFULL PERMISSIONS FULL;
// DEFINE FIELD IF NOT EXISTS title ON TABLE post TYPE string;
// DEFINE FIELD IF NOT EXISTS published_at ON TABLE post TYPE option;
}
fn down() -> String {
// REMOVE TABLE IF EXISTS post;
}
}
The #[field(thing)] annotation tells the macro that id is the SurrealDB record ID (which gets special treatment — it’s omitted from DDL since SurrealDB manages it automatically, and primary_key() returns it).
Other field attributes:
AttributeEffect#[field(thing)]Mark as SurrealDB record ID#[field(record = “other_table”)]Typed record link#[field(name = “different_name”)]Override the SurrealDB field name#[field(default = “expr”)]Set a DEFAULT expression in DDL#[field(value = “expr”)]Set a VALUE expression (computed field)#[field(flexible)]Allow flexible typing#[field(skip)]Omit from schema generation#[field(ty = “custom_type”)]Override the inferred SurrealQL type
The type inference is currently string-based — the macro inspects the Rust type string for keywords like “String”, “i64”, “Option” and maps them to SurrealQL types. This is pragmatic and works for the common cases, but richer type analysis (using syn to walk the actual type tree) is on the roadmap.
Real-World Patterns
Pattern 1: Repository Layer
Wrap Somnia in a repository that encapsulates query logic:
struct PostRepository {
client: SomniaClient,
}
impl PostRepository {
pub async fn find_by_title(&self, title: &str) -> Result> {
self.client.query(
&Post::table()
.select(Post::all())
.filter(Post::title().eq(title))
.limit(50)
).await
}
pub async fn find_recent(&self, since: &str) -> Result> {
self.client.query(
&Post::table()
.select(Post::all())
.filter(ident("published_at").greater_than(since))
.order_desc(ident("published_at"))
.limit(20)
).await
}
pub async fn create(&self, post: &Post) -> Result> {
self.client.query(
&Post::table()
.create()
.content(post)
.returning(Returning::After)
).await
}
}
Pattern 2: Schema Verification in CI
Generate the DDL during CI and compare it against production:
#[test]
fn schema_ddl_matches_expected() {
// This test fails if someone changes a struct without updating the snapshot
insta::assert_snapshot!(Post::up());
}
Or, more ambitiously, connect to a staging SurrealDB instance, run the generated DDL, and verify no errors:
#[tokio::test]
async fn schema_up_is_valid_surrealql() {
let client = SomniaClient::connect("mem://", "", "", "", "").await.unwrap();
client.raw(&Post::up()).await.expect("Schema DDL should be valid");
client.raw(&Post::down()).await.expect("Schema teardown should be valid");
}
Pattern 3: Composable Filter Functions
Build reusable filter predicates:
fn published_since(cutoff: &str) -> impl Expr {
ident("published_at").greater_than(cutoff)
}
fn title_contains(phrase: &str) -> impl Expr {
Post::title().contains(phrase.to_string())
}
fn authored_by(author: &str) -> impl Expr {
Post::title().eq(author.to_string())
}
// Compose them
let filter = published_since("2025-01-01")
.and(title_contains("SurrealDB").or(authored_by("Vikram")));
let query = Post::table()
.select(Post::all())
.filter(filter)
.limit(25)
.to_surrealql();
Pattern 4: Batch Operations
Group multiple statements into a single round-trip:
use somnia::query::Batch;
let batch = Batch::new()
.statement(
Post::table()
.create()
.content(&post1)
.returning(Returning::After)
)
.statement(
Post::table()
.create()
.content(&post2)
.returning(Returning::After)
)
.statement(
Post::table()
.select(Post::all())
.filter(Post::title().eq("batch test"))
);
// Execute as one network call; results come back in order
let results = client.query(&batch).await?;
The CLI: Project Tooling Beyond the Library
The somnia CLI binary is modeled on diesel-cli. Install it with:
cargo install somnia-cli # from crates.io
brew tap vbasky/somnia && brew install somnia # or via Homebrew
The workflow for a new migration:
1. Scaffold a migration directory. Run somnia migration generate create_posts_table, which creates:
migrations/2025-06-15-142300_create_posts_table/
├── up.surql
└── down.surql
2. Edit up.surql with your DDL:
DEFINE TABLE IF NOT EXISTS post SCHEMAFULL;
DEFINE FIELD IF NOT EXISTS title ON TABLE post TYPE string;
...
3. Edit down.surql with the reverse:
REMOVE TABLE IF EXISTS post;
4. Apply it. Run somnia migration run.
5. Check what’s applied. Run somnia migration list — it prints ✓ 2025-06-15-142300_create_posts_table.
6. Iterate. somnia migration redo reverts and re-applies (good for development); somnia migration revert reverts the most recent.
The CLI connects to the same SurrealDB instances your application uses, tracking migration state in the _somnia_migrations table. Multiple developers can run migrations against the same database safely — the tracking table ensures each migration is applied exactly once.
What Somnia Isn’t (Yet)
I want to be upfront about the current limitations. Somnia is at version 0.4.1, and there are things it doesn’t do:
No ODM / Active Record. Somnia is a query builder and thin client, not an object-document mapper. There’s no change tracking, no dirty-checking, no post.save(). You write queries, they execute, you get results. A persistence ergonomics layer could be built on top, but that’s not Somnia’s job.
No Graph Traversal Helpers. SurrealDB supports path expressions like person->authored->post, but Somnia doesn’t yet have typed graph traversal in SELECTs. This is P1 on the roadmap.
No Live Queries. SurrealDB’s LIVE SELECT for real-time updates isn’t modeled in the builder yet. P3.
No Transaction Wrappers. BEGIN TRANSACTION / COMMIT / CANCEL aren’t wrapped. You can use raw() for this. P2.
String-Based Type Inference. The derive macro does simple string matching on Rust type names to infer SurrealQL types. Complex generics, custom types, and newtype wrappers may need explicit #[field(ty = “…”)] annotations. Richer type analysis is P1.
Not Production-Battle-Tested at Scale. The covered features work correctly, but this is a young project (0.4.x). Comprehensive CI validates every commit, but you should test thoroughly for your use case.
Getting Started in 5 Minutes
Add Somnia to your project:
[dependencies]
somnia = "0.4"
serde = { version = "1", features = ["derive"] }
tokio = { version = "1", features = ["full"] }
surrealdb = "2" # required for the SurrealDB client
Define a record and connect:
use somnia::prelude::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Serialize, Deserialize, SurrealRecord)]
#[table("user")]
struct User {
#[field(thing)]
id: Thing,
name: String,
email: String,
}
#[tokio::main]
async fn main() -> somnia::Result<()> {
// Connect to SurrealDB (in-memory for testing, ws:// for real)
let client = SomniaClient::connect(
&std::env::var("SURREAL_ENDPOINT")
.unwrap_or_else(|_| "mem://".into()),
&std::env::var("SURREAL_USER").unwrap_or_else(|_| "root".into()),
&std::env::var("SURREAL_PASS").unwrap_or_else(|_| "root".into()),
&std::env::var("SURREAL_NS").unwrap_or_else(|_| "test".into()),
&std::env::var("SURREAL_DB").unwrap_or_else(|_| "test".into()),
).await?;
// Set up the schema
client.raw(&User::up()).await?;
// Insert a user
let user = User {
id: Thing::new("user", "1"),
name: "Ada Lovelace".into(),
email: "ada@computing.org".into(),
};
client.insert(&User::table().insert(vec![&user])).await?;
// Query it back
let users: Vec = client.query(
&User::table()
.select(User::all())
.filter(User::name().eq("Ada Lovelace"))
).await?;
assert_eq!(users.len(), 1);
println!("Found user: {:?}", users[0]);
// Clean up
client.raw(&User::down()).await?;
Ok(())
}
Why I Built This
I’ve spent too many late nights debugging SurrealQL strings. A missing backtick here, a wrong quote character there, a UUID that got parsed as math — these are the bugs that slip past code review because they look correct at a glance. They only surface when data hits a specific shape, or when a user types a special character, or when you upgrade SurrealDB and the parser gets stricter.
Somnia is my answer to that frustration. It moves SurrealDB interaction errors from runtime to compile time. It makes the Rust compiler your first line of defense against broken queries. And it treats the database schema as a first-class citizen of your codebase — derivable, testable, and version-controlled.
The project is Apache-2.0 licensed, actively maintained, and tested against SurrealDB 3.x. If you’re building Rust applications on SurrealDB, I’d love for you to try it, open issues, and contribute.
Links
GitHub: github.com/vbasky/somnia
Crates.io: crates.io/crates/somnia
Documentation: docs.rs/somnia
Homebrew: brew tap vbasky/somnia && brew install somnia
Have you hit query-string bugs in production? What’s your approach to type-safe database access in Rust? Drop a comment — I’d love to hear your war stories and ideas for what Somnia should tackle next.

Measure, Don’t Guess: Building viser, a Content-Adaptive Video Encoding Optimizer in Rust

Vikram Bhaskaran — Fri, 05 Jun 2026 04:48:02 GMT

Most video on the internet is encoded with a one-size-fits-all bitrate ladder that wastes bandwidth on simple content and starves complex content. viser is a bet that we can do better — by measuring every video instead of guessing.
There is a small act of quiet waste happening millions of times a day. Somewhere a streaming service takes a video — a news anchor reading the headlines, a fast-cut action sequence, a grainy noir thriller — and runs all of them through the same table of bitrates. The anchor gets far more bits than her flat, static frame could ever use. The thriller gets nowhere near enough, and its shadows dissolve into blocky mud. The table doesn’t care. It was written once, in 2010, and it has been copied ever since.
viser is built on a simple conviction: every video deserves to be looked at. Not guessed at from a lookup table, not predicted from a thumbnail — actually encoded, actually measured, and given the bitrate ladder that wrings the most perceptual quality out of every single bit. The name blends vision and optimizer (and nods to French viser, “to aim”). It is fifteen Rust crates, roughly 8,000 lines, and a belief that the right answer is worth the work of finding it.
This is the story of how it works, why the hard parts are hard, and what it took to make a tool whose every numerical claim you can reproduce yourself.
The Problem: One Ladder Does Not Fit All
Every streaming service faces the same decision, over and over: at what resolutions and bitrates do I store this video so a player can adaptively switch between rungs as the network changes?
The easy answer is a fixed bitrate ladder — one table of (resolution, bitrate) rungs applied to everything. Apple’s original 2010 HLS recommendation defined ten rungs from 64 kbps up to 8564 kbps at 1080p, and most of the industry copied it. It is simple, and it is wasteful, because content does not have uniform complexity:
A fixed ladder simultaneously overspends on the anchor and starves the thriller. The talking head reaches transparent quality at 1.5 Mbps and 720p; the action sequence needs 8 Mbps and 1080p to reach the same perceived quality. Netflix proved the fix in 2015 — per-title encoding — and demonstrated 20–30% bitrate savings. The catch is that doing it correctly is a measurement problem, and measurement problems are where good intentions quietly go wrong.
viser exists to do that measurement correctly, at four levels of granularity, for three codecs, with results you can reproduce.
Four Granularities, One Framework
“Content-adaptive encoding” is not one algorithm. It is a family, distinguished by how finely you adapt:
All four sit on the same foundation, and all four had to be right not just in shape but in number. A convex hull that is “roughly right” produces a ladder that is roughly right, which means bandwidth you can measure on a bill. The whole craft of the project is in turning “roughly” into “exactly.”
When There’s No Single Right Answer to Diff Against
Here is what makes this kind of tool intimidating to build: the output of a per-title analysis is a judgment — a set of rungs — and reasonable engineers disagree about the last rung. There is no canonical perfect-ladder binary you can run alongside yours and demand identical output. So what do you validate against?
The breakthrough is to realize that the oracle is the published science, decomposed into pieces that each have a knowable right answer:
The convex hull has a correct answer. Given a set of (bitrate, VMAF) points, the upper Pareto frontier is a deterministic geometric object. Andrew’s monotone chain produces it in O(n log n), exactly. You can pin it against hand-computed hulls: empty input, a single point, all-collinear points, interior points that must be removed, unsorted input.
BD-Rate — the Bjøntegaard delta that says “codec A needs X% less bitrate than codec B for equal quality” — is a published numerical method with reference implementations and known outputs. Feed in the canonical test vectors; demand the same percentage back.
The reference ladders are public. Netflix and Apple’s fixed ladders are documented rung-for-rung. viser carries them as comparison baselines, and a test asserts they are reproduced exactly — because a tool that can’t even echo a fixed ladder has no business computing an adaptive one.
The encode and measurement bottom out at ffmpeg and libvmaf. viser orchestrates them rather than reimplementing them, so the oracle for “did this encode hit the target bitrate” is ffmpeg’s own output, probed back and checked.
This decomposition is the heart of the whole thing. You cannot diff a judgment — but you can diff every input to the judgment. Get the hull exactly right, get BD-Rate exactly right, get the bitrate measurement exactly right, and the judgment that rides on top of them is correct by construction. Every module in viser had to answer one question: what could I be objectively wrong about? — and then a test was written to pin it. That discipline is what turns an intimidating problem into a tractable one.
Architecture: Fifteen Crates in Four Layers
viser is fifteen crates arranged in four strict layers, each crate with a single responsibility:
Foundation viser-ffmpeg FFmpeg/FFprobe wrapper — encode, probe, cache
viser-quality VMAF / PSNR / SSIM / SSIMULACRA2 / Butteraugli
viser-encoding Shared config, preset mapping, temp cleanup
viser-checkpoint SHA-256 resumable state

Core viser-hull Convex hull (Pareto frontier) + BD-Rate
viser-ladder Rung selection with crossover enforcement
viser-shot Shot/scene detection (FFmpeg scdet)
viser-complexity Spatial / temporal / DCT analysis
Pipelines viser-pertitle Whole-video ladder
viser-pershot Per-shot + Trellis bit allocation
viser-persegment Segment-level CRF adaptation
viser-contextaware Device-specific ladders
Application viser-cli The clap binary
viser-compare Browser comparison player
viser-chart R-D curve charts (plotters)
The dependency graph only ever points down. A pipeline crate composes foundation and core crates; it never reaches sideways into another pipeline. viser-pershot is allowed to depend on viser-pertitle (a shot is just a short title), but viser-pertitle knows nothing about shots.
This is where Rust’s crate model earns its keep. Every cross-module call is an explicit, typed, public API — there is no accidental reach-around between optimization logic and ffmpeg-shelling logic, because the crate boundary makes it impossible to write one by accident. Fifteen shallow crates, not a deep inheritance tree. Wide and flat, where each piece is small enough to hold in your head and test in isolation.
Race the Encodes, Walk the Budget
A full per-title search is 5 resolutions × 9 CRF values × 3 codecs = 135 trial encodes. Serially, that's an afternoon. All at once, it's a fork bomb that thrashes the machine into swap and produces slower wall-clock time than a careful subset. The art is in racing the encodes while walking the resource budget — and viser models that bound explicitly:
// Bound concurrent encodes to num_cpus/2 — encoders are already
// internally threaded, so oversubscribing thrashes the cache.
let permits = Arc::new(Semaphore::new(num_cpus::get() / 2));
let trials = trial_matrix.into_iter().map(|trial| {
let permit = permits.clone();
async move {
let _slot = permit.acquire().await?; // backpressure, not a fork bomb
let encoded = encode(&trial).await?;
let vmaf = measure_vmaf(&source, &encoded).await?;
Ok(RdPoint::new(trial, encoded.bitrate, vmaf))
}
});
The semaphore is the entire concurrency story, and it is visible. There is no hidden task that outlives its parent, no channel that deadlocks if a consumer panics. Tokio’s structured concurrency means a failed trial propagates and cancels the rest rather than orphaning them.
And because every long analysis is checkpointed — each completed trial hashed by its (resolution, codec, CRF, preset) config under a SHA-256 key — a crash three hours into a 135-trial sweep resumes from trial 130, not trial zero. The same hash is the cache invalidation: change the preset, the hash changes, stale results are silently ignored. Long, expensive work that survives a crash is not a luxury; it's what makes the whole "measure everything" philosophy practical instead of fragile.
What Makes This Genuinely Hard
It would be dishonest to pretend the language makes this correct. Rust makes it safe; the domain makes it hard, and no language helps with the domain. The genuinely difficult parts are worth naming, because they are also the parts that make the project interesting:
Resolution crossovers are content-dependent. Each resolution traces an R-D curve that flattens at a quality ceiling — 480p tops out around VMAF 98 because downsampling destroys high-frequency detail that no number of bits can recover; 1080p reaches 99.7. The bitrate at which 1080p overtakes 720p on the hull is not a constant. For a talking head it happens low; for film grain it happens high, because the lower resolution hasn’t hit its ceiling yet. The hull finds the crossover automatically — but only if you compute it across all resolutions jointly, a subtlety that quietly produces a plausible-but-wrong ladder if you get it backwards.
VMAF is expensive, and the cost compounds. Every one of those 135 trials needs a VMAF measurement, and VMAF runs on the CPU through libvmaf. The honest accelerations — n_subsample=5 to measure every fifth frame, fast presets for the search and slow presets only for final delivery, eliminating dominated resolutions after a single probe CRF — are all approximations, each trading accuracy for time. So viser exposes them as flags rather than baking in a guess. The trade is the user's to make, with eyes open.
Per-shot allocation is a global optimization, not a local one. You cannot optimize each shot independently — that spends your whole budget on the first complex shot. The correct method is Trellis/Lagrangian allocation: find the single quality-per-bit slope λ at which the sum of per-shot bitrates hits your global target, then assign every shot the operating point at that slope. It’s a one-dimensional search over λ with a convex-hull lookup per shot inside the loop, and every failure mode (empty hull, single shot, identical shots, duration weighting) earned its own test.
HDR breaks the metric. libvmaf is SDR-centric. An HDR source measured with an SDR model produces a number that looks fine and means nothing. viser detects HDR from probe metadata and refuses it by default, gated behind an explicit --allow-hdr for people who know they're doing something best-effort. Honesty as a feature, refusal as a kindness.
None of these are language problems. They are video problems that any serious implementation has to get right — which is exactly why the 170+ tests exist: the compiler cannot catch a wrong crossover bitrate, so a human writes the assertion that does.
The Heart: Convex Hull and BD-Rate
The core of the whole project is about 540 lines in viser-hull, and it is the part I trust most, because it is the part with the least ambiguity. The upper hull is textbook Andrew's monotone chain, specialized to (bitrate, quality) space:
/// Upper convex hull of R-D points: the Pareto frontier where no other
/// configuration achieves higher quality at the same or lower bitrate.
fn upper_hull(mut points: Vec) -> Vec {
points.sort_by(|a, b| a.bitrate.total_cmp(&b.bitrate));
let mut hull: Vec = Vec::new();
for p in points {
// Pop while the last turn is clockwise or collinear — those
// middle points are dominated and must leave the frontier.
while hull.len() >= 2 && cross(hull[hull.len() - 2], hull[hull.len() - 1], p) >= 0.0 {
hull.pop();
}
hull.push(p);
}
hull
}
total_cmp instead of partial_cmp is a small thing that matters: a stray NaN from a failed measurement must not silently corrupt the sort, and Rust forces you to confront float ordering rather than let < quietly do the wrong thing. The crate is #![forbid(unsafe_code)]; nothing in hull geometry needs raw pointers.
BD-Rate sits beside it — the cubic fit over log(bitrate) versus quality, integrated over the overlapping quality range, returned as a percentage. It has the most published reference outputs, and therefore the most assertive tests: minimum-point handling, negative efficiency (when codec A is worse), non-overlapping ranges, and the near-singular matrices the cubic fit produces on pathological inputs.
Validation: Reproduce the Known Before Trusting the Novel
The test suite is 170+ tests across the workspace, and its philosophy is “pin everything that has a knowable answer”:
cargo test -p viser-hull # convex hull, BD-rate — 24 tests
cargo test -p viser-ladder # rung selection, crossover, savings — 19 tests
cargo test -p viser-pershot # Trellis Lagrangian allocation — 12 tests
cargo test -p viser-complexity # complexity + screen-content detection — 21 tests
cargo test -p viser-ffmpeg # probe, encode args, metadata adapter — 26 tests
The coverage maps directly onto “what could each module be wrong about”:
Hull: empty, single point, interior-point removal, unsorted input, per-codec separation.
BD-Rate: minimum points, negative efficiency, non-overlap, singular matrices, cubic fit.
Trellis: empty, single shot, duration weighting, identical shots, empty-hull fallback, λ-search bounds.
Ladder: empty, zero rungs, bitrate/VMAF filters, max-VMAF cap, sorted output, the Netflix and Apple reference ladders reproduced exactly, savings computation.
Screen content: slides classified 90%, natural video 0%, code-capture 70%, empty input.
The whole workspace runs in about half a second, which means the tests run on every save, which means the math stays pinned. Fast tests are honest tests — the ones too slow to run get run never, and the modules they protect drift. The single most valuable test in the suite is the cheapest one: the assertion that viser reproduces the Netflix and Apple fixed ladders rung-for-rung. Reproduce the known before you trust the novel.
Safety and Honesty
The pure-math and orchestration crates are unsafe-free, the workspace denies the usual footgun lints, and the whole thing is edition = "2024" on a pinned MSRV of 1.88. There is no unsafe in hull geometry, ladder selection, or Trellis allocation, because none of those domains need it — they are arithmetic and Vecs.
What Rust really buys here isn’t raw speed — the bottleneck is ffmpeg, not the optimizer. It’s the elimination of a class of silent wrongness. A missing bitrate can’t quietly default to 0 and distort a frontier; Option, exhaustive match, and the absence of an implicit zero-value force every "what if this is missing" to become a decision the compiler makes you write down. In a tool whose entire value is the trustworthiness of a number, that guarantee is the foundation everything else stands on.
The Frontier: Beyond the Obvious
The four methods are the spine, but the most exciting part of viser is where it goes past the textbook:
Two extra perceptual metrics — SSIMULACRA2 and Butteraugli run in viser-quality alongside VMAF, PSNR, and SSIM. VMAF is excellent, but it is one model; cross-checking against a second perceptual metric catches the cases where VMAF is over-confident.
Screen-content detection. Slides, code screencasts, and UI captures behave nothing like camera footage — static frames, razor-sharp edges, DCT energy concentrated completely differently. viser-complexity classifies content from spatial/temporal/DCT heuristics so the pipeline can choose a strategy instead of treating a slide deck like a football game.
A pure-Rust probe engine — revelo. By default viser reads metadata with ffprobe, but an optional engine called revelo (--features revelo) extracts resolution, frame rate, codec, color space, and dynamic range with no external binary on the path at all. More on what that unlocks below.
Audio-aware ladder budgets. per-title analyze detects the source audio bitrate and reserves it, so a "3 Mbps" rung is 3 Mbps of total stream, not 3 Mbps of video plus whatever audio happens to cost.
A browser comparison player that shows reference and encoded side-by-side with a per-frame VMAF timeline — because at some point a human has to look, and a number on a chart is not the same as seeing the artifact disappear.
What I’d Want You to Take From This
If there’s a lesson worth carrying out of this project, it’s smaller and more hopeful than “rewrite it in Rust”:
When you can’t diff the answer, diff the inputs to the answer. A judgment isn’t testable; the geometry, the numerical method, and the measurement underneath it all are. Decompose until every piece has a knowable right answer, and the judgment becomes correct by construction. This turns intimidating, “expert-only” problems into ones an ordinary careful engineer can ship.
Reproduce the known before trusting the novel. The cheapest test can be the most valuable one. If your tool can’t echo a published reference, it has no authority to compute something new.
Make your approximations into flags, not assumptions. Every speed/accuracy trade you hide is a guess you’re making on someone else’s behalf. Expose it, and the trade becomes their informed choice.
Measure, don’t guess. The whole project is one sentence long: look at the actual video, run the actual encodes, trust the number you can reproduce. It is more work than a lookup table. It is also just better, and the difference shows up on a bandwidth bill and in a viewer’s eyes.
The Result
viser is three things at once, depending on how you reach for it:
# A CLI: compute an optimal ladder across codecs and resolutions
viser per-title analyze -i video.mp4 \
--codecs libx264,libsvtav1 \
--resolutions 480p,720p,1080p \
--preset veryfast --parallel 4 -o analysis.json
# Then deliver the selected rungs as final encodes
viser per-title deliver --analysis analysis.json \
--output-dir delivery --mode capped-crf --manifest delivery/manifest.json
# A probe — with no ffprobe required
viser inspect probe video.mp4 --probe-engine revelo
# And a set of Rust crates you can compose yourself
Four optimization methods, three codecs, five quality metrics, fifteen crates, about 8,000 lines of unsafe-free Rust, under a permissive BSD-2-Clause license — and every numerical claim pinned to a test you can run in half a second.
cargo install viser-cli --features revelo
Most video on the internet is still encoded by a table written before the iPhone had a Retina display. It doesn’t have to be. Every video deserves to be looked at — and now there’s a tool that does exactly that.
viser is open source at github.com/vbasky/viser.
One More Thing: How revelo Changes the Equation
Everything above quietly leans on a dependency I’ve barely mentioned: to set up its trial matrix, viser has to read the video first — resolution, frame rate, codec, color space, dynamic range. The default way to do that is to shell out to ffprobe, which means a native binary has to be installed, found on PATH, version-matched, and trusted to parse a container the same way every time. For a tool whose whole promise is reproducibility, leaning on an external parser is the one soft spot in the foundation.
revelo is the answer to that — a pure-Rust probe engine built directly into viser. Enable it with --features revelo, ask for it with --probe-engine revelo, and viser reads container and stream metadata itself, in process, with nothing external on the path:
cargo install viser-cli --features revelo
viser inspect probe video.mp4 --probe-engine revelo # no ffprobe anywhere
That single change rewrites the equation in a few ways:
One fewer native dependency. The metadata path becomes pure Rust. Distribution gets dramatically simpler — closer to “download one binary and run it” than “install this toolchain first.”
Reproducibility all the way down. The probe is no longer a black box whose behavior depends on which ffprobe build happens to be installed. The same input yields the same parse, pinned by the same fast tests as the rest of the workspace (codec mapping, color transfer, pixel format, frame-rate formatting all have their own assertions).
Memory safety on untrusted input. Parsing arbitrary media containers is exactly the kind of byte-wrangling that has produced a decade of CVEs in C parsers. Doing it in unsafe-free Rust turns a whole class of buffer bugs into compile-time impossibilities — which matters the moment you point the tool at a file you didn't create.
ffprobe stays the friendly default, because it is battle-tested and ubiquitous. But revelo is the direction of travel: a video optimizer that, end to end, depends on nothing but itself — and answers you can reproduce on any machine, with no toolchain to install first. The measurement philosophy, finally, reaches all the way down to the first byte.
Additional Reading
If this piece sparked your curiosity, the science docs go a layer deeper:
Per-Title Encoding — convex hulls, rate-distortion optimization, and how a ladder gets selected
Per-Shot Encoding — shot detection and Trellis constant-slope bit allocation
Segment-Level CRF — closed-loop CRF tuning with complexity analysis
Quality Metrics — VMAF, PSNR, SSIM, and BD-Rate, demystified
Netflix: Per-Title Encode Optimization — the 2015 post that started it all

Porting 300,000 Lines of C++ and Perl to Rust: A Dual-Oracle Media Metadata Engine

Vikram Bhaskaran — Fri, 29 May 2026 03:36:40 GMT

How I replaced both a 25-year-old C++ media analyzer and a 30-year-old Perl EXIF tool with a pure Rust library — matching their output byte-for-byte and tag-for-tag, while making it faster, safer, and installable with cargo install.
The Problem
Every video and photo file carries hidden metadata. Codec profiles, color spaces, HDR luminance values, frame rates, audio channel layouts, bitrates, encoder settings, GPS coordinates, lens specifications, maker-note camera settings. For decades, two tools have been the undisputed standards:
MediaInfoLib (C++): extracts container/codec metadata from ~200 media formats
ExifTool (Perl): extracts EXIF/maker-note metadata from thousands of camera models
Both work. Both are painful to integrate.
MediaInfoLib
Try adding it to a modern project. First you build it — ./Configure && make && make install, which pulls in 10+ system library dependencies (zlib, libcurl, libmms, libzen, libtinyxml2, and more). Each has its own version constraints. If you're on macOS, Homebrew helps. On Windows, you're in for a bad afternoon. In CI, every container image needs the full toolchain. Cross-compile? Good luck.
Integration options:
Shell out as a subprocess, capture XML, parse it back. Lose type safety, pay the serialization round-trip, error handling is “parse the stderr string.”
Link via C ABI. Manage a native dependency across platforms. The C ABI was designed for C++ consumption — char* output buffers, manually-managed handles, integer stream kind discriminants you must get exactly right.
ExifTool
Written in Perl. Amazing tool — 30,000+ tags, 50+ camera makes. But embedding it means shipping a Perl runtime, managing CPAN dependencies, and accepting that your parser lives in a scripting language with no memory safety guarantees.
Both codebases touch attacker-controlled binary data. 200,000 lines of C++ and Perl parsing crafted MP4 boxes and EXIF IFDs. Every file is a potential exploit vector.
This is exactly the problem Rust was built to solve.
The Challenge: Not One Oracle, But Two
Let me be clear about what “port” means here. This is not a wrapper. It’s a complete from-scratch reimplementation in Rust that produces:
Byte-identical XML output to MediaInfoLib — every field name, decimal separator, bucket, field ordering identical.
95–100% tag parity with ExifTool — for 14 camera vendors, validation against the exiftool binary shows exact tag name and value matches on real camera files.
Two reference tools’ worth of metadata, in one pure-Rust crate.
Why Both?
They solve different halves of the same problem:
Architecture: From Virtual Classes to Function Pointers
The most visible design change: instead of a C++ class hierarchy, revelo uses a flat table of 180 function pointers:
pub fn table() -> [fn(&mut MediaFile) -> bool; 180] {
[
parse_wav, // WAV
parse_mp4, // MP4/MOV
parse_mkv, // Matroska
parse_hevc, // HEVC
parse_jpeg, // JPEG + EXIF
parse_exif, // EXIF IFD walker
// ... 174 more
]
}
Every parser is fn(&mut MediaFile) -> bool. No traits, no vtables, no subclassing. Each parser is standalone, independently testable, and trivially parallelizable.
Race + Walk
All 180 parsers race across CPU cores using rayon’s work-stealing thread pool:
pub fn detect(bytes: &[u8]) -> Option bool> {
table()
.par_iter()
.find_first(|&&parser| {
let mut fa = MediaFile::new(bytes);
parser(&mut fa) // cheap detection: peek magic bytes
})
.copied()
}
Detection is just checking magic bytes — WAV checks for RIFF, MP4 checks ftyp box, JPEG checks 0xFFD8. Each candidate finishes in microseconds. On an 8-core machine, the entire 180-format race completes in under a millisecond.
Once detected, the winner runs again for full extraction. Two passes: parallel sprint, then single-threaded deep parse.
What Makes This Hard: Media Parsing Is Inconsistent by Design
Media files aren’t cleanly-specified formats read by well-behaved parsers. They’re the product of 30 years of competing standards committees, vendor extensions, and “it compiles, ship it” engineering.
MP4/MOV is nominally a tree of boxes, each with a 4-byte size and 4-byte type. But some boxes use 0 as a size meaning "rest of file." Others use 1 as a flag meaning "the real size is in the next 8 bytes" (64-bit extended size). The stsd sample description box is a container whose children vary by codec type — avcC for H.264 stores an SPS/PPS blob, hvcC for H.265 stores a VPS/SPS/PPS combo with completely different bit-packing, dvcC for Dolby Vision has its own versioning, and esds for AAC contains a recursive ISO 14496-1 descriptor hierarchy (DecoderConfigDescriptor → DecoderSpecificInfo → SLConfigDescriptor) that must be parsed depth-first.
Matroska/WebM uses EBML elements with variable-length integer-encoded sizes. The size encoding is elegant — the number of leading zeros in the first byte tells you how many bytes follow — but decoding it requires bit-level manipulation. A single Segment element can span terabytes with nested seek indexes, track definitions, chapters, tags, attachments, and CueSeekPositions all as sibling elements.
MPEG-TS is a flat stream of 188-byte packets. Each packet has a PID identifying its stream, but the PAT (PID 0) and PMTs must be parsed to know which PID carries video, which audio, and which subtitles. The PMT itself contains nested descriptors — some simple (4 bytes, language code), some complex (AC-3 descriptor with bitstream mode and channel configuration). You can’t rely on the PMT appearing at a fixed position; you have to scan continuously.
RIFF (AVI/WAV) uses little-endian chunk sizes, which is fine until you encounter a WAV file with an odd-sized fmt chunk and need to handle the word-alignment padding byte exactly as the C++ code does. Or a Broadcast Wave bext chunk whose specification includes a field that is literally "372 bytes of reserved space, must be zeroed."
Bitstream-level parsers are the worst. HEVC SPS parsing requires extracting log2_max_pic_order_cnt_lsb_minus4 from a ue(v) (unsigned exponential Golomb-coded integer), which itself must be decoded bit-by-bit: count leading zeros, read that many data bits, add 1, subtract 1. H.264 has a similar scheme with completely different parameter names and semantics. AV1 uses a different OBU (Open Bitstream Unit) structure entirely. Each codec has its own bit-packing conventions, and none of them align with byte boundaries.
Each container format requires a fundamentally different walking strategy. I chose not to abstract over them — every container walks its native on-disk structure directly. The output model is the same StreamCollection — they all write into the same data structure — but the walk code maps directly to the format specification.
The Porting Process
I started every parser as a transliteration of the C++ source, then iteratively refined through differential testing:
Find sample files — real-world media, sometimes generated with specific encoder settings for obscure formats.
Run the oracle — mediainfo --Output=XML sample.file. Save ground truth.
Read the C++ source — File_* class, Header_Parse() and Data_Parse() flow, transliterate field-by-field into Rust.
Compare output — cargo run -p revelo-diff -- sample.file.
Fix discrepancies — mismatched field names, wrong byte order, wrong bit offsets, wrong value formatting. Every mismatch is a bug.
Commit — once --strict reports byte-equal.
Some parsers came together in hours. The MP4 container parser — with dozens of sub-box types — took weeks. The hardest bugs were the ones where output looked right but the bytes didn’t match: a field rendered as "48000" instead of "48 000", or appearing in the wrong XML element order because the C++ code Fill()-s fields in a different sequence than alphabetical.
The FileAnalyze Engine
The engine wraps the raw buffer and provides the read API every parser uses. I kept MediaInfoLib’s C++ API naming — get_b1 through get_b8 for big-endian, get_l1 through get_l8 for little-endian, get_c4 for four-character codes, bs_begin()/bs_end() for bitstream mode — so I could compare the Rust port against the C++ original side-by-side without mental translation.
The read API is comprehensive — everything from single-byte peeks to 80-bit IEEE 754 extended precision floats (AIFF format uses these):
Big-endian: get_b1 through get_b8, get_b16, peek_b*, skip_b*
Little-endian: get_l1 through get_l8, get_l16, with peek/skip variants
Floats: get_bf4, get_bf8, get_bf10 (80-bit extended precision)
Bitstream: bs_begin()/bs_end() bracket bit-aligned reads with get_s1 through get_s8 (MSB-first)
4CC: get_c4 reads 4 bytes as big-endian u32, traces as human-readable ASCII
Raw: read_raw(n), peek_raw(n), peek_raw_at(at, n) for absolute reads
Underrun behavior follows the C++ semantics exactly: any read exceeding the buffer sets a truncated flag, advances the cursor to EOF, and returns zero. No panics, no crashes. Real media files are often truncated, and a parser that panics is useless in production.
The ExifTool Layer: 14-Camera Maker-Note Depth
This is where revelo diverges from being “just a MediaInfoLib port.”
The core crate is BSD-2-Clause and uses hand-written clean-room maker-note tables for standard EXIF/GPS/Interop tags. For camera-maker-note depth on par with ExifTool, an opt-in exiftool-tables feature pulls in tag tables generated from ExifTool's source:
cargo install revelo-cli --features exiftool-tables
Because those tables are derived from ExifTool, the revelo-exiftool-tables crate is GPL/Artistic, not BSD. The licensing is deliberately isolated: the default build stays BSD-2-Clause; only a build that opts in takes on the copyleft.
Camera Coverage
Validated by revelo-exif-diff, a harness that diffs revelo's output against the exiftool binary tag-for-tag — the same way revelo-diff validates against mediainfo.
EXIF Architecture
EXIF parsing in revelo has two layers:
The IFD walker (tags.rs): full IFD chain traversal (IFD0 → ExifIFD → GPS IFD → InteropIFD → IFD1), 170+ standard tag names, GPS coordinates computed as decimal degrees, LensSpec formatting from multi-field combinations.
The format detector (jpeg.rs): a curated MediaInfo-style parser that extracts a streamlined set of EXIF fields using MediaInfoLib-compatible names (Recorded_Date, Encoded_Hardware_CompanyName). Both layers write to the same Exif stream with non-conflicting keys, so consumers can query whichever naming convention they prefer.
Byte-for-Byte Validation
My validation standard: exact — not “close enough.” Two differential test harnesses:
# MediaInfo oracle
$ cargo run -p revelo-diff -- --strict video.mp4
Oracle: 312 lines, Rust: 312 lines
Only in oracle: 0 Only in rust: 0 Common: 312
✅ Byte-equal!
# ExifTool oracle
$ cargo run -p revelo-exif-diff Canon_40D.jpg
TOTAL: 45/45 exiftool EXIF/maker-note tags matched (100%)
The MediaInfo harness uses a full LCS (Longest Common Subsequence) O(n×m) table diff in --strict mode. When it reports zero differences, the outputs are byte-equal — every character, every space, every angle bracket identical.
The ExifTool harness runs exiftool -G1 -s against the same file, collects all EXIF/maker-note tag names, and counts how many revelo also produces (matched by tag name, case-insensitive). On 19 of 20 sample files across 20 camera models and brands, revelo achieves 100% tag parity.
Safety and Correctness
Zero unsafe blocks. #![deny(unsafe_code)] across all 17 crates.
This isn’t philosophical. Media files are attacker-controlled inputs. A parsing library will encounter maliciously crafted files:
Heap overflows from crafted box sizes
Out-of-bounds reads in bitstream parsing
Use-after-free in box tree manipulation
In Rust, the type system and borrow checker eliminate these classes of bugs entirely. If it compiles, it doesn’t have a buffer overflow. Vec::resize with a user-supplied size will either succeed or panic — no silent heap corruption. Bounds-checked indexing means every read is validated. The Option type eliminates null pointer dereferences.
The HDR Frontier
Beyond porting, I extended revelo with HDR format detection that even MediaInfoLib v26.05 doesn’t cover:
HDR10+ (SMPTE ST 2094–40): parsed from HEVC SEI payload type 4 and AV1 metadata OBUs
Dolby Vision RPU: NAL unit type 62 parsing with L1/L5/L8 metadata layers
SL-HDR1 (ETSI TS 103 433): three SEI payload types
HLG/PQ container signalling: CICP transfer_characteristics detection
Dolby Atmos: E-AC-3 dependent substreams and TrueHD MAT frames
Eclipsa Audio (IAMF): full OBU-level parsing with scalable channel layouts
HDR classification needs data from both the container layer (MP4 colr boxes, MKV Colour elements) and the elementary bitstream (HEVC/AV1/AVC SPS VUI colour info). revelo emits colour_primaries, transfer_characteristics, matrix_coefficients, HDR_Format, MasteringDisplay_Luminance, and MasteringDisplay_ColorPrimaries from both layers.
Dolby AC-4: Frame header parsing with CBR/VBR signaling, substream table (up to 16 substreams with independent sample rates and channel modes), IMS (Immersive Stereo) bit, JOC (Joint Object Coding) object count, and dialnorm/loudness extraction.
Lessons Learned
1. Flat is better than deep. The C++ virtual class hierarchy made sense for the original developers — adding a format meant subclassing File__Analyze and overriding Header_Parse() and Data_Parse(). But that design carries costs: virtual dispatch overhead on every call, complex initialization ordering, state sprawl from inherited member variables, and testing isolation difficulty. My flat function table is simpler, faster (no vtables), parallelizable, and each parser is a pure function from MediaFile to bool with no shared mutable state.
2. Parallel detection is a win that falls out of the architecture. I didn’t set out to parallelize format detection. It became trivial once I had a flat table of independent functions. The C++ sequential if-else chain is O(n) in the number of formats, and n keeps growing. With rayon, it’s one parallel iteration — essentially constant time on any multi-core machine.
3. Byte-level correctness requires byte-level testing. You cannot reason your way to bit-exact output compatibility. The differential harness catches bugs that are invisible to functional testing: field ordering differences (the C++ code Fill()-s fields in one order while my code does another, breaking schema validation), whitespace differences in XML attribute ordering, value formatting differences (integer rendered as "48000" vs "48 000").
4. Container-native is the right call. Tempting as it is to build a unified “media stream” abstraction, each container stores metadata differently — boxes (MP4), EBML elements (MKV), RIFF chunks (AVI), PES packets (MPEG-TS). Forcing them into a common model would lose information or require constant exceptions. My approach — walk each container through its own structure, write to the same stream model — strikes the right balance. The walk code maps directly to the format spec; the output is unified.
5. Two oracles beat one. Adding ExifTool validation alongside MediaInfo validation exposed a different class of bugs — tag name mismatches, missing maker-note sub-IFDs, GPS coordinate sign errors. Each oracle validates a different dimension of the output, and passing both gives confidence that neither silent brokenness remains.
The Result
Revelo ships as 17 workspace crates with zero native dependencies. The cdylib crate is a drop-in replacement for libmediainfo.so/.dylib/.dll — same C ABI, same output, completely different implementation underneath. The exiftool-tables optional crate brings ExifTool-derived maker-note depth for 14 camera vendors, validated at 95–100% tag parity.
# Drop-in for libmediainfo:
MediaInfo_Handle handle = MediaInfo_New();
MediaInfo_Open(handle, "video.mp4");
const char* xml = MediaInfo_Inform(handle, 0); // byte-identical output
The project compiles with a single cargo build from a clean checkout, runs on any platform with a Rust toolchain (including wasm32-unknown-unknown), and distributes via cargo install. No ./Configure, no system library dependencies, no Perl runtime.
Repository: github.com/vbasky/revelo — BSD-2-Clause (core) · GPL/Artistic (optional exiftool-tables).
cargo install revelo-cli
revelo --xml video.mp4 # MediaInfoLib-compatible XML
revelo photo.jpg --json # JSON with EXIF/maker-note tags
revelo --json --features exiftool-tables # Full ExifTool-grade maker-note depth
Or as a Rust library:
// One-liner: detect, parse, extract tags
let meta = revelo::Metadata::from_file("photo.jpg").unwrap();
for (key, value) in meta.general() {
println!("{key} = {value}");
}
for (key, value) in meta.exif() {
println!("{key} = {value}");
}
No native dependencies. No system libraries. No Perl runtime. cargo build from a clean checkout.