Documentation
¶
Overview ¶
Package rank turns per-change measurements into honest ranks, one metric at a time (ADR 0001, docs/metrics.md#ranking-rules): a bootstrap gives each subject a 95% interval for its value and for its rank, subjects below the metric's minimum sample are listed and not ranked, and AUC scores a metric against hand labels for the validity check.
It is part of the functional core: plain functions over the clusters a ranked metric hands it (metrics.RankInput), with no I/O. How a replicate is drawn depends on the metric's Kind:
- Ratio metrics, such as durable@1, resample each subject's clusters (its changes) with replacement: a cluster bootstrap (Efron and Tibshirani, An Introduction to the Bootstrap, 1993). Lines inside a change are not independent, so resampling lines would give falsely narrow intervals.
- Sum metrics, such as simplifying@1, resample the window's week blocks, the same blocks for every subject: a block bootstrap (Künsch, "The Jackknife and the Bootstrap for General Stationary Observations", Ann. Statist., 1989). A total depends on how many changes landed, which resampling a subject's own clusters would hold fixed: for a pure count every replicate would return the observed count.
Every draw is reproducible. The seed derives from the metric, scope and window (Seed), each random stream from the seed, the generator is math/rand/v2's PCG, and the bounded draw is this package's own, so the same inputs give the same intervals on every platform and Go release.
Index ¶
- Constants
- func AUC(pos, neg []float64) float64
- func AUCDiffInterval(id domain.MetricID, pos, neg [][2]float64) (lo, hi float64)
- func AUCInterval(id domain.MetricID, pos, neg []float64) (lo, hi float64)
- func Intervals(id domain.MetricID, scope domain.Scope, w domain.Window, kind Kind, ...) []domain.RankInterval
- func Seed(id domain.MetricID, scope domain.Scope, w domain.Window) uint64
- type Cluster
- type Kind
- type Options
- type Separation
Constants ¶
const ( BarAUC = 0.7 BarLabels = 50 )
The bar a ranked metric's labels clear (product.md, ADR 0014): an AUC of at least BarAUC on at least BarLabels of its scored labels. validate says whether labels clear it; label asks until each metric has BarLabels.
const ( VerdictTooFew = "cannot tell yet" VerdictNone = "does not separate meaningful from routine changes" VerdictReverse = "separates in reverse (meaningful changes score lower)" VerdictWeak = "separates weakly, below the bar" VerdictBeyond = "separates, beyond change size" VerdictLikeSize = "separates, but no better than change size" )
The verdicts, in docs/metrics.md's words.
Variables ¶
This section is empty.
Functions ¶
func AUC ¶
AUC returns the Mann-Whitney area under the ROC curve: the probability that a value from pos exceeds one from neg, counting ties as one half. It is the Mann-Whitney U statistic over len(pos)·len(neg) pairs (Mann and Whitney, Ann. Math. Statist., 1947), which equals the area under the empirical ROC curve (Hanley and McNeil, "The Meaning and Use of the Area under a Receiver Operating Characteristic (ROC) Curve", Radiology, 1982): 1 when every positive scores above every negative, 0.5 at chance, 0 when the order is reversed. stillhere validate reports it per metric version, with pos the values of changes labelled meaningful and neg those labelled routine (docs/metrics.md#validity-checks).
Values are ordered as cmp.Compare orders them, and the pairs are counted in integers, so the result is exact up to its one division. With no pair to compare (pos or neg empty) there is no evidence either way, and AUC returns 0.5, never NaN; callers report the sample size next to it. The inputs are not modified.
func AUCDiffInterval ¶
AUCDiffInterval returns a 95% percentile interval of AUC(a) − AUC(b), where a and b are two scores of the same labelled changes: each element of pos is one meaningful change's pair {a, b}, and of neg one routine change's. Each of 1,000 replicates resamples the changes once, the meaningful and the routine apart as AUCInterval does, and takes both AUCs over the very changes it drew, so the two stay paired: their correlation on the same changes narrows the interval, which two separate intervals ignore. It is the bootstrap form of the paired comparison of two AUCs on one sample (DeLong, DeLong and Clarke-Pearson, "Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves", Biometrics 44(3), 1988). validate compares a ranked metric with change size alone by it: the metric says more than size only when the interval lies above 0 (Separation.Verdict). The pairs are sorted first and the draws come from a stream of their own, derived from id, so the interval depends on the scores alone, as AUCInterval's does. With pos or neg empty there is no pair to compare, and both ends are 0. The inputs are not modified.
func AUCInterval ¶
AUCInterval returns a 95% percentile interval of AUC(pos, neg) for the metric id: 1,000 bootstrap replicates, each resampling pos and neg with replacement and apart, so every replicate keeps as many meaningful and as many routine labels as the sample (a bootstrap stratified by class), and the interval runs from the 25th to the 975th of the sorted replicates, as the rank intervals do (Efron and Tibshirani, An Introduction to the Bootstrap, 1993, ch. 13). DeLong, DeLong and Clarke-Pearson's variance (Biometrics, 1988) is the analytic alternative; the bootstrap needs no normal approximation, which a sample of 50 labels and an AUC near 1 strain. The draws come from a stream derived from id alone, and resample pos and neg sorted, so the interval depends on the scores alone, not on the order of the labels file's rows: the same labels give the same interval in any order, on every platform and Go release. With pos or neg empty there is no pair to compare, and both ends are 0.5, as AUC is. The inputs are not modified.
func Intervals ¶
func Intervals(id domain.MetricID, scope domain.Scope, w domain.Window, kind Kind, data map[domain.Subject][]Cluster, opt Options) []domain.RankInterval
Intervals returns one rank interval per subject in data, sorted by subject (Kind, then Key), for metric id in scope over window w. It returns nil when data is empty.
A subject is eligible with at least opt.MinN clusters, and at least one. Each of B replicates (opt.Replicates, or 1,000) resamples the eligible subjects' clusters as their Kind says (see the package comment) and ranks them by value, the largest first. Tied values share the best rank they cover, as in "1, 1, 3" (standard competition ranking), so a tie never ranks anyone below the subjects it ties with. Lo and Hi are the ⌈0.025·B⌉-th and ⌈0.975·B⌉-th of the sorted replicate values, the 25th and the 975th of 1,000 (the percentile interval of Efron and Tibshirani, 1993, ch. 13), and RankLo and RankHi are the same order statistics of the subject's ranks.
A subject that is not eligible is listed and not ranked: Estimate and N are set, it takes no rank in any replicate, and Lo, Hi, RankLo and RankHi are 0. Below the minimum sample a bootstrap understates its own uncertainty (a ratio over one cluster always gets a zero-width interval), so no interval is reported.
Estimate is Σ Num over the subject's clusters for Sum, and Σ Num / Σ Den for Ratio, where a zero Σ Den gives 0 so that no NaN reaches a report. Values are compared exactly: sums of counts are exact in float64, and equal ratios of counts round to the same float64, so they tie. The result depends on each subject's set of clusters, not on their order, and a Ratio subject's value interval depends on its own clusters only. Two Sum subjects with the same totals in every block get the same intervals, because every replicate draws the same blocks for both; two Ratio subjects with the same clusters get intervals that agree only up to Monte Carlo error, because each draws from its own stream. Intervals ranks every subject it is given: leaving bots out is the caller's job (docs/metrics.md#ranking-rules).
Intervals panics on input that only a bug in the ranked metric can produce, because such input would otherwise give plausible but wrong intervals: a Kind other than Ratio and Sum and, for Sum, an empty window, an Options.Blocks other than the number of blocks in w, or a cluster whose Block is outside [0, Blocks), whether or not its subject is eligible. Too few blocks hide a count's uncertainty, and too many inflate it. The options are checked even when data is empty, so a misconfigured metric fails on its first call.
func Seed ¶
Seed derives the bootstrap seed from the metric, scope and window: the first 8 bytes, big-endian, of a SHA-256 over a length-prefixed encoding of the metric's name and version, the scope's kind and key, and the window's two instants. The same inputs give the same seed on every platform and Go release, a window's location does not change it, and a new metric version, another scope or another window gives an unrelated one.
Types ¶
type Cluster ¶
type Cluster struct {
Change string // landing SHA or PR id
Num float64 // the change's numerator
Den float64 // Ratio only
// Block is Sum only: the window's 7-day block the change landed in,
// counted back from the window's end, which is Now for every Sum metric:
// w.Week(t) for a change at time t (domain.Window.Week, the one
// definition of the rule). Block k holds the landings in
// [Until − 7(k+1) days, Until − 7k days), half-open like the window, so a
// landing at time t is in block ⌈(Until − t) / 7 days⌉ − 1, and 0 is the
// last 7 days. It must be in [0, Options.Blocks).
Block int
}
Cluster is one change's contribution to one subject's value. Changes, not lines, are resampled, because lines inside a change are not independent.
type Options ¶
type Options struct {
Replicates int // B; 0 means the default, 1000
MinN int // eligibility: clusters per subject, from the metric's definition
// Blocks is Sum only: the number of 7-day blocks the window holds,
// ⌈(Until − Since) / 7 days⌉, the oldest one short when the window is
// not a whole number of weeks: 13 for 90 days. It is w.Weeks()
// (domain.Window.Weeks), and Intervals checks it against the window.
Blocks int
}
Options tunes the bootstrap. The ranked metric supplies them (metrics.RankInput), because eligibility is part of its definition.
type Separation ¶
type Separation struct {
AUC, Lo, Hi float64 // AUC and AUCInterval of the metric's scores
N int // the changes labelled meaningful or routine that it scored
Size float64 // AUC of change size alone over the same changes
DiffLo, DiffHi float64 // AUCDiffInterval of the metric's score and size: AUC less Size
}
Separation is what validate finds for one ranked metric on one set of labels: its AUC with the 95% interval and n, and the AUC of change size alone over the very changes it scored, with the paired interval of the difference. Verdict and Result say what it means, by the one rule docs/metrics.md#validity-checks states, for a lead's labels and for a published reference alike.
func (Separation) Result ¶
func (s Separation) Result() string
Result is the verdict with the numbers it rests on, as a metric's Validity line publishes it: "does not separate meaningful from routine changes: AUC 0.42 (0.30–0.55) on 76, while change size alone gives 0.82". The two verdicts that compare with size add the difference and its paired interval, which decide between them.
func (Separation) Verdict ¶
func (s Separation) Verdict() string
Verdict is the first of these that holds: fewer than BarLabels changes cannot tell yet; an interval that holds 0.5 does not separate; one below 0.5 separates in reverse; an AUC under BarAUC separates weakly, below the bar; and a metric at the bar separates beyond change size when the paired interval of its AUC less size alone's lies above 0, else no better than change size, since a metric that grows with size can separate the classes through size alone. So "separates" is never said of an interval that holds chance, and "beyond change size" never of a difference that could be 0.
Source Files