Standing on good work
Forensic surface comparison already has excellent, hard-won tools — many open-source, and several that Verity builds directly on. So why a new one? Not because the others are wrong, but because the field’s own reviews identified gaps that no single existing tool closes. Verity is the attempt to close all of them at once.
Built on the CSAFE research program
Verity did not invent the science of forensic surface comparison — it operationalizes and deploys it. The methods below, developed largely within CSAFE and at Iowa State, are the shoulders Verity stands on. Our contribution is the production layer: one calibrated, explainable pipeline, an open codec, a hosted API, and reproducible validation on top of this research program.
Automatic bullet matching
Verity's bullet-land pipeline descends directly from the CSAFE/Iowa State automatic-matching method — surface signatures, alignment, and a learned similarity score, validated on the community's bullet benchmarks.
Congruent Matching Cells
The cell-counting idea at Verity's core is Song's CMC. Verity generalizes it to Congruent Matching Regions so one pipeline spans striated and impressed marks, rather than a bespoke method per mark type.
Public benchmark data
Verity is validated against the data the community built and shares — Hamby 252/173 bullets from the open NIST NBTRD, Fadul consecutively-manufactured cartridge cases from CSAFE-ISU's open scan repository, and screwdriver toolmarks from the open tmaRks dataset.
Non-affiliation. Verity is an independent project and is not affiliated with, sponsored by, or endorsed by CSAFE or Iowa State University. References to their published research denote scientific lineage and validation targets only. A formal collaboration is something we would propose, not something that exists today.
| Approach | Striated | Impressed | One method | Calibrated LR | Attribution | Open |
|---|---|---|---|---|---|---|
Trained examiner the status quo | yes | yes | yes | no | partial | no |
bullet lands · Hare et al. 2017 | yes | no | no | partial | partial | yes |
breech faces · Song 2013 | no | yes | no | partial | yes | yes |
Chumbley score striated toolmarks | yes | no | partial | partial | no | partial |
Deep-learning match raw similarity score | yes | yes | partial | no | no | no |
Verity one calibrated method | yes | yes | yes | yes | yes | yes |
✓ yes · ◐ partial / with caveats · ✕ no. A high-level summary; each tool is excellent at what it was built for.
Measured against the specialists, on their home turf
A capability table is a claim; this is the measurement — and it is deliberately the hardest kind. Each specialist is run on the data it was built for; Verity is held to the same scans under a source-disjoint split (no barrel, slide, or tool appears in both train and test), with the exact scorer and protocol labeled on each row. Every row is scored per dataset and never pooled across makes, on two axes: AUC (can it tell same-source from different-source apart?) and the forensic Cllr (is the weight of evidence well-calibrated? 0 is perfect, 1 is useless). These are real results — including where Verity trails.
Bullet landsstriated
vs bulletxtrctr — random-forest matchscore · Hare et al. 2017
| Study | Verity AUC | bulletxtrctr AUC | Verity Cllr | bulletxtrctr Cllr |
|---|---|---|---|---|
Hamby-252 46 KM · 549 KNM Verity = production CMR-1D (diag_contrast) | 1.000 | 1.000 | 0.113 | 0.064 |
Prince George's County PD (PGPD) Beretta 30 KM · 405 KNM Verity = production CMR-1D (diag_contrast) | 0.999 | 1.000 | 0.273 | 0.171 |
Same scans, same barrel-disjoint folds, both scorers. The trained specialist leads on both sets — expected: Verity's bullet contribution is the calibrated, bounded, deployable LR layer, not a better matcher.
Cartridge breech facesimpressed
vs CMC (cmcR) — Congruent Matching Cells · Song 2013
| Study | Verity AUC | CMC (cmcR) AUC | Verity Cllr | CMC (cmcR) Cllr |
|---|---|---|---|---|
Fadul 10-slide 10 KM · 180 KNM Verity = deployed CMR-2D — frozen benchmark cartridge-v1 (fold mean) | 0.922 | 1.000 | 0.398 ±0.202 | 0.194 |
Verity quotes the frozen public benchmark cartridge-v1 (fold mean), per the cartridge quoting policy in docs/headline-numbers.md; cmcR is the measured slide-disjoint specialist baseline on the same 190 Fadul pairs, and it leads on this set.
Screwdriver toolmarksstriated
vs Chumbley U — non-random U-statistic · toolmaRk
| Study | Verity AUC | Chumbley U AUC | Verity Cllr | Chumbley U Cllr |
|---|---|---|---|---|
tmaRks 56 tool-edges 3,530 KM · 163,802 KNM Verity = deployed CMR-1D — frozen benchmark toolmark-v1 (fold mean) | 0.943 | — | 0.330 ±0.047 | — |
Ames Lab screwdrivers 15 KM · 103 KNM Verity = global 1-D CCF — proof scorer, not the deployed CMR-1D | 0.807 | 0.608 | 0.813 | 0.957 |
Two distinct datasets: the deployed toolmark figure is the frozen tmaRks benchmark (toolmark-v1, 56 tool edges). Ames Lab (7 tools, 118 pairs) is a small proof set the whitepaper calls a weak benchmark for any method, and its Verity-side scorer is the naive global 1-D CCF, not the deployed pipeline; it is kept because Chumbley U was run here only on Ames Lab.
Validated on further studies
Verity — production CMR-1D (diag_contrast), barrel-disjoint fold means — on bullet studies where no specialist baseline was run here.
| Study | Barrels | AUC | Cllr | Cllr min |
|---|---|---|---|---|
Phoenix PD (Ruger P-95) — known barrels (NBTRD) 24 KM · 252 KNM | 8 | 0.972 | 0.354 | 0.105 |
Hamby (2009) Barrel Set 173 — full (NBTRD) 46 KM · 549 KNM | 10 | 0.971 | 0.338 | 0.158 |
Higher AUC and lower Cllrare better; the stronger figure in each pair is highlighted. Every row shares Verity’s preprocessing-and-calibration pipeline, and the scorer behind each Verity figure is labeled on its row: bullet rows use the production CMR score; the cartridge and tmaRks rows quote the frozen public benchmarks, so they match the open benchmark and the method page exactly; the Ames Lab row is a non-deployed proof scorer kept for the Chumbley comparison. The trained specialists lead on their home sets — bulletxtrctr on bullets, cmcR on Fadul — as expected: Verity’s contribution is the calibrated, bounded, deployable LR layer, not a better matcher. We report the losses alongside the wins on purpose; every figure here traces to a protocol-labeled row in the numbers registry.
The four gaps
Two decades of review — the National Research Council (2009) and PCAST (2016) — and recent work by Cuellar et al. (2024) converge on the same shortfalls.
One method, not many
Bullets, cartridges, and toolmarks have each grown their own hand-engineered pipeline — the field reinvents feature extraction per mark type. Verity generalizes the Congruent Matching Cells idea to arbitrary marks (Congruent Matching Regions): striated and impressed flow through one pipeline, differing only in the registration group.
A likelihood ratio, not a score
A random-forest matchscore, a CMC count, a U-statistic — none is a weight of evidence. The reportable answer should be a calibrated likelihood ratio with a characterized cost (Cllr), the standard from forensic speaker comparison (Brümmer & du Preez, 2006). Verity reports exactly that, bounded to what the reference data can support.
Glass-box, not black-box
Recent deep networks beat classical methods on raw discrimination — but emit an uncalibrated score, no region-level attribution, and an architecture that invites source-code litigation. Verity keeps the decision a monotone, bounded transform of the score and shows the exact regions that drove it, so the result is auditable no matter how the score was computed.
Honest about scope
Cuellar et al. (2024) found that noforensic firearm discipline has a characterized error rate. Verity doesn’t claim a universal accuracy; it reports Cllr on a named reference population and states its scope plainly — and refuses to answer outside the data it was calibrated on.
How Verity responds to Cuellar et al. (2024)
Cuellar, Vanderplas, Luby & Rosenblum showed that the black-box studies underpinning firearms identification do not establish a characterized error rate. We think they are right — and Verity is built to supply precisely what they found missing. This is not a rebuttal; it is the concrete answer their critique calls for.
The black-box studies behind firearms identification do not establish a characterized error rate.
Verity reports a Cllr — a characterized calibration cost — on a named reference population, instead of an implied universal accuracy.
Inconclusive and missing responses are counted in ways that make the studies look more accurate than they are.
Verity never forces a categorical call: it reports a bounded likelihood ratio (an empirical cap on what the reference data can support), so weak evidence reads as a weak LR — not a discarded inconclusive.
Claims of validity outrun the data behind them.
Verity states its scope plainly and declines to answer outside the population it was calibrated on — a glass-box, monotone map, not a verdict.
One of the critique’s authors co-authored bulletxtrctr, which Verity benchmarks against — the same community, working toward the same standard of honesty. Verity does not claim to characterize the error rate of human examination, which remains unknown; it characterizes the cost of its own calibrated number on a stated reference.
What the courts are asking for
The same demands now arrive from the bench. Three primary sources frame what a forensic comparison method must show before its conclusions reach a jury:
Daubert v. Merrell Dow
Daubert v. Merrell Dow Pharmaceuticals, 509 U.S. 579 (1993) made trial judges gatekeepers of expert evidence, weighing testability, peer review, known or potential error rates, and controlling standards.
FRE 702, as amended
Federal Rule of Evidence 702, amended December 1, 2023, makes explicit that the proponent must show it is more likely than not that the testimony reflects a reliable application of reliable methods — and that the expert’s opinion stays within what the methodology can support.
Abruquah v. Maryland
In Abruquah v. Maryland, 484 Md. 64 (2023), Maryland’s highest court held that a firearms examiner could not testify that crime-scene bullets were fired from a particular gun — the underlying studies did not support an unqualified source identification.
None of this is legal advice, and no court has evaluated Verity itself. It is the standard Verity is engineered against: a calibrated likelihood ratio with a characterized cost (Cllr) on a named reference population, bounded to what that reference can support — never an unqualified “match.”
Built on the field, not against it
Verity generalizes CMC, borrows its calibration from speaker detection, and validates on the very datasets the community built — Hamby, NIST, and others. It does not replace the examiner; it hands them a transparent, calibrated number they can stand behind. That is what was missing, and why Verity exists.