benefit

package
v0.3.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 21, 2026 License: MIT Imports: 10 Imported by: 0

Documentation

Overview

Package benefit is the Jev benefit benchmark's durable core (ADR-001 amendment A1): the frozen ground truth, the run record, the grader, and the statistics. It answers a different question from the selection gate — not "did the agent pick the right route" but "is the decision better, and at what latency, when the same model is forced through evaluate instead of answering directly". Nothing here talks to a model or to Jev: a record holds observations, and every grade and metric is recomputed from them, so a run can be regraded without being rerun.

Index

Constants

View Source 
const (
	FaultNoEvaluate = "no_evaluate_call" // A2: the forced call never came
	FaultInvalid    = "invalid_request"  // A2: the server refused the request
	FaultShape      = "wrong_shape"      // A2: not one question of the labelled primitive
	FaultUnreadable = "unreadable_jev"   // A2: Jev's result held no answer for the question
	FaultUnparsed   = "unparsed_answer"  // A0/A1: no {"decision": ...} in the final answer
	FaultStalled    = "model_stalled"    // any arm: a model turn never finished
)

Faults: why a rep produced no gradable decision. A fault is counted as incorrect — the benchmark measures the path users run, and a path that yields no decision did not decide — and is tallied so the accuracy of well-formed attempts can be read separately.

View Source 
const (
	ArmDirect = "A0"
	ArmAuto   = "A1"
	ArmForced = "A2"
)

Arms. A0 answers directly with no tools; A1 is the selection harness's own agent loop; A2 forces evaluate and takes Jev's raw answer as final.

View Source 
const (
	StatusComplete = "complete"
	StatusPartial  = "partial"
)
View Source 
const GraderVersion = 1

GraderVersion changes whenever a grade could come out differently from the same observations. A record keeps the version that graded it.

View Source 
const SchemaVersion = 1

SchemaVersion of the benefit record. It is a separate contract from the selection run record: the two are never read by the same code.

Variables

This section is empty.

Functions

func Write 

func Write(dir string, rec Record) (string, error)

Write persists rec under dir as benefit-YYYYMMDD-<sha8>.json and never replaces a file: evals/results is append-only.

Types

type ArmDef 

type ArmDef struct {
	System     string `json:"system"`
	Tools      string `json:"tools"`
	ToolChoice string `json:"tool_choice"`
	Turns      int    `json:"turns"`
	Terminal   string `json:"terminal"`
}

ArmDef records what an arm was, in the run's own words: prompts change the experiment, so they are part of the record, not of the code alone.

type Call 

type Call struct {
	Tool      string `json:"tool"`
	Arguments string `json:"arguments,omitempty"` // evaluate: verbatim
	Valid     *bool  `json:"valid,omitempty"`     // evaluate: the server accepted it
	Rejection string `json:"rejection,omitempty"`
	Output    string `json:"output,omitempty"` // evaluate: Jev's raw result
}

Call is one tool call as the model made it.

type Decision 

type Decision struct {
	Bool   *bool    `json:"bool,omitempty"`
	Option string   `json:"option,omitempty"`
	Level  *int     `json:"level,omitempty"`
	Raw    *float64 `json:"raw,omitempty"`
}

Decision is a normalized terminal decision. Exactly one of Bool, Option, Level is set; Raw keeps the probability or score it was derived from.

type Effect 

type Effect struct {
	Scenarios int     `json:"scenarios"`
	Points    float64 `json:"points"`
	CILow     float64 `json:"ci_low"`
	CIHigh    float64 `json:"ci_high"`
	Wins      int     `json:"wins"`   // A2 right, A0 wrong
	Losses    int     `json:"losses"` // A0 right, A2 wrong
	Ties      int     `json:"ties"`
	McNemarP  float64 `json:"mcnemar_p"`
}

Effect is the paired A2−A0 accuracy effect over the same scenarios. Points is in proportion units (0.10 = ten percentage points). Wins, Losses, Ties and McNemarP use each scenario's primary rep; the interval is a paired bootstrap over scenarios, stratified by family, resampling reps within a scenario.

type Environment 

type Environment struct {
	Platform    string `json:"platform"`
	Hardware    string `json:"hardware,omitempty"`
	ServerBuild string `json:"server_build,omitempty"`
	ServerNotes string `json:"server_notes,omitempty"` // quantization, context, backend
	Concurrency int    `json:"concurrency"`
}

Environment is what a latency number is valid for. Latency measured here is a fact about this machine, server build, and configuration — never a universal property of the model.

type Grade 

type Grade struct {
	ScenarioID string    `json:"scenario_id"`
	Arm        string    `json:"arm"`
	Rep        int       `json:"rep"`
	Family     string    `json:"family"`
	Decision   *Decision `json:"decision,omitempty"`
	Correct    bool      `json:"correct"`
	Fault      string    `json:"fault,omitempty"`
	// A1 only, when a well-shaped evaluate call was answered: what Jev said,
	// apart from what the agent finally concluded.
	JevDecision *Decision `json:"jev_decision,omitempty"`
	JevCorrect  *bool     `json:"jev_correct,omitempty"`
}

Grade is the derived verdict on one rep. Negative scenarios and excluded ambiguous ones carry no label and get no grade.

func GradeAll 

func GradeAll(reps []Rep, labels Labels) []Grade

GradeAll grades every rep that has a label, in record order.

func GradeRep 

func GradeRep(r Rep, lb Label, rules Rules) Grade

GradeRep derives one rep's grade from its observations.

type Label 

type Label struct {
	Type      string   `json:"type"`
	Condition string   `json:"condition,omitempty"`
	Expected  any      `json:"expected,omitempty"`
	Options   []string `json:"options,omitempty"`
	Levels    int      `json:"levels,omitempty"`
	Min       int      `json:"min,omitempty"`
	Max       int      `json:"max,omitempty"`
}

Label is one scenario's ground truth. Expected is a bool for noul and an option for choice; score carries an inclusive level range instead.

func (Label) AnswerFormat 

func (lb Label) AnswerFormat() string

AnswerFormat is what A0 and A1 are asked to end with, so a direct answer and a Jev answer are graded in the same normalized shape.

func (Label) DecisionBrief 

func (lb Label) DecisionBrief() string

DecisionBrief states the decision every arm owes, in the same words for every arm. It is appended to the scenario prompt; only the sentence that follows it (AnswerFormat or ToolShape) differs between arms.

func (Label) ToolShape 

func (lb Label) ToolShape() string

ToolShape is A2's instruction: one question, of the labelled primitive, in a shape the grader can read back.

type Labels 

type Labels struct {
	Version        int               `json:"version"`
	Note           string            `json:"note,omitempty"`
	Rules          Rules             `json:"rules"`
	StabilityPanel []string          `json:"stability_panel"`
	Labels         map[string]Label  `json:"labels"`
	Excluded       map[string]string `json:"excluded,omitempty"`
	SHA256         string            `json:"sha256,omitempty"`
}

Labels is labels.json. SHA256 is of the file as loaded and travels with every record, so a regrade knows exactly which truth the run was held to.

func LoadLabels 

func LoadLabels(path string) (Labels, error)

LoadLabels reads and validates labels.json. A label that cannot be graded would silently score every arm wrong, so it is refused here.

func (Labels) IDs 

func (l Labels) IDs() []string

IDs returns the labelled scenario ids, sorted.

func (Labels) Reps 

func (l Labels) Reps(scenarioID string) int

Reps is how many repetitions a scenario gets: fixed before the run, never chosen from what a run showed.

type Latency 

type Latency struct {
	N           int     `json:"n"`
	TotalMedian float64 `json:"total_median_ms"`
	TotalP90    float64 `json:"total_p90_ms"`
	ModelMedian float64 `json:"model_median_ms"`
	ModelP90    float64 `json:"model_p90_ms"`
	JevMedian   float64 `json:"jev_median_ms"`
	JevP90      float64 `json:"jev_p90_ms"`
	// RetriedReps counts reps whose Jev time includes a retry backoff. They
	// stay in the percentiles above — the pre-registered statistic — and are
	// disclosed so a slow tail can be told apart from a slow Jev.
	// UnknownAttemptReps counts reps where an accepted call carries no attempt
	// count: they may hide a backoff, and RetriedReps says nothing about them.
	RetriedReps        int `json:"retried_reps"`
	UnknownAttemptReps int `json:"unknown_attempt_reps"`
}

type LatencyDelta 

type LatencyDelta struct {
	Scenarios  int     `json:"scenarios"`
	MedianMS   float64 `json:"median_ms"`
	CILow      float64 `json:"ci_low_ms"`
	CIHigh     float64 `json:"ci_high_ms"`
	SpeedRatio float64 `json:"a0_over_a2_speed_ratio"` // median A0 total / median A2 total
}

LatencyDelta is A2−A0 total time, paired by scenario. It may be negative.

type Negatives 

type Negatives struct {
	Reps                int     `json:"a1_reps"`
	UnnecessaryEvaluate int     `json:"unnecessary_evaluate_reps"`
	InvalidWhenMisused  int     `json:"invalid_evaluate_calls"`
	A0MedianMS          float64 `json:"a0_total_median_ms"`
	A1MedianMS          float64 `json:"a1_total_median_ms"`
}

Negatives is what the scenarios where evaluate is the wrong mechanism show. A1−A0 latency here is the cost of the whole tool environment and the routing it induced, not of listing evaluate.

type Rate 

type Rate struct {
	Correct int     `json:"correct"`
	N       int     `json:"n"`
	Rate    float64 `json:"rate"`
}

Rate is a proportion with its counts visible: 3/4 and 30/40 are not the same evidence.

type Record 

type Record struct {
	SchemaVersion int               `json:"schema_version"`
	Mode          string            `json:"mode"` // always "benefit"
	Run           Run               `json:"run"`
	Arms          map[string]ArmDef `json:"arms"`
	Labels        Labels            `json:"labels"` // snapshot: the truth this run was held to
	Reps          []Rep             `json:"reps"`
	GraderVersion int               `json:"grader_version"`
	Grades        []Grade           `json:"grades"`
	Summary       Summary           `json:"summary"`
}

Record is one benefit run. Grades and Summary are a snapshot of what the run concluded, kept for audit; Regrade recomputes both from Reps+Labels.

func NewRecord 

func NewRecord(run Run, arms map[string]ArmDef, labels Labels, reps []Rep, runErr error) Record

NewRecord assembles the record for a run and grades it. A run that ended early, for any reason, is partial.

func ReadDir 

func ReadDir(dir string) ([]Record, error)

ReadDir reads every benefit-*.json under dir, oldest run first.

func (Record) Complete 

func (r Record) Complete() bool

Complete reports whether every planned rep was observed. A partial run is evidence for debugging and nothing else.

func (Record) Regrade 

func (r Record) Regrade() Record

Regrade recomputes every grade and metric from the recorded observations and the snapshotted labels, under the current grader.

type Rep 

type Rep struct {
	ScenarioID string `json:"scenario_id"`
	Kind       string `json:"kind"` // positive | negative | ambiguous
	Arm        string `json:"arm"`
	Rep        int    `json:"rep"`
	// Sequence is the rep's position in the whole run; ArmOrder is the order
	// the arms ran in for this scenario and rep.
	Sequence int      `json:"sequence"`
	ArmOrder []string `json:"arm_order"`

	Calls       []Call  `json:"calls"`
	FinalAnswer string  `json:"final_answer,omitempty"` // the model's closing text, if it gave one
	Spans       []Span  `json:"spans"`
	TotalMS     float64 `json:"total_ms"`
	Tokens      Tokens  `json:"tokens"`
	Turns       int     `json:"turns"`
	// Stalled: a model turn outran the harness's per-turn budget, so the rep
	// has no answer. What came before the stall is kept.
	Stalled bool `json:"stalled,omitempty"`
}

Rep is one arm of one scenario, once: observations only. Whether it was correct is a grade (Grade), derived from these plus the labels.

func (Rep) Route 

func (r Rep) Route() []string

Route is the tools a rep called, in order.

type Rules 

type Rules struct {
	NoulTrueAt         float64 `json:"noul_true_at"`
	ScoreRounding      string  `json:"score_rounding"`
	LatencyTieMS       float64 `json:"latency_tie_ms"`
	BootstrapResamples int     `json:"bootstrap_resamples"`
	BootstrapSeed      uint64  `json:"bootstrap_seed"`
	ArmOrderSeed       uint64  `json:"arm_order_seed"`
	PrimaryReps        int     `json:"primary_reps"`
	PanelReps          int     `json:"panel_reps"`
}

Rules are the benchmark's pre-registered conventions. They classify for this benchmark only: NoulTrueAt is not a product probability policy.

type Run 

type Run struct {
	Date          time.Time   `json:"date"`
	GitSHA        string      `json:"git_sha,omitempty"`
	Model         string      `json:"model"`
	Endpoint      string      `json:"endpoint"`
	Temperature   float64     `json:"temperature"`
	MaxTurns      int         `json:"max_turns"`
	JevRoute      string      `json:"jev_route,omitempty"` // typesafe | openrouter; never a credential
	CorpusSHA256  string      `json:"corpus_sha256"`
	Environment   Environment `json:"environment"`
	WarmUpCalls   int         `json:"warm_up_calls"`
	MaxJevCalls   int         `json:"max_jev_calls"`   // the cap the run was started under
	PlannedJevMax int         `json:"planned_jev_max"` // worst case the plan could have made
	JevCalls      int         `json:"jev_calls"`
	ExpectedReps  int         `json:"expected_reps"`
	WallMS        float64     `json:"wall_ms"`
	Status        string      `json:"status"`
	Failure       string      `json:"failure,omitempty"`
}

Run is what the caller knows about a run.

type ScenarioOutcome 

type ScenarioOutcome struct {
	ScenarioID string  `json:"scenario_id"`
	Family     string  `json:"family"`
	AccDelta   float64 `json:"accuracy_delta"` // mean over reps, A2−A0
	LatDeltaMS float64 `json:"latency_delta_ms"`
	Accuracy   string  `json:"accuracy"` // better | same | worse
	Latency    string  `json:"latency"`  // faster | same | slower
}

ScenarioOutcome places one scenario on the two axes. Accuracy and latency are never folded into a single winner.

type Span 

type Span struct {
	Kind string  `json:"kind"` // model | evaluate
	MS   float64 `json:"ms"`
	// Attempts is the upstream requests an accepted evaluate call took. An
	// accepted call takes at least one, so 0 is "the server did not say" —
	// a lost stats line, or a rep recorded before the count existed — and
	// must never be read as "one request, no retry" (ADR-005 §4).
	Attempts int `json:"attempts,omitempty"`
}

Span is one timed step of a rep, in the order it happened.

type Stability 

type Stability struct {
	PanelScenarios int            `json:"panel_scenarios"`
	Disagreeing    map[string]int `json:"disagreeing"`
}

Stability is the fixed panel's verdict on run-to-run variation: panel scenarios whose reps did not all grade the same, per arm.

type Summary 

type Summary struct {
	Accuracy map[string]map[string]Rate `json:"accuracy"` // arm → family|"all" (M1)
	Faults   map[string]map[string]int  `json:"faults"`   // arm → fault
	// WellFormedA2 is A2 accuracy over reps that reached a Jev answer.
	WellFormedA2 Rate              `json:"a2_well_formed"`
	JevGain      map[string]Effect `json:"jev_gain"` // family|"all" (M2)
	// OracleRegret (M3): mean of max(A0,A2) − A1 on primary reps. Hindsight
	// only; it never defines the right route and never relabels a scenario.
	OracleRegret          float64 `json:"oracle_routing_regret"`
	OracleRegretScenarios int     `json:"oracle_routing_regret_scenarios"`
	// A1 reps where a well-shaped evaluate call was answered: what Jev said
	// versus what the agent finally concluded.
	A1JevRaw     Rate               `json:"a1_jev_raw"`
	A1AfterJev   Rate               `json:"a1_final_after_jev"`
	Latency      map[string]Latency `json:"latency"`       // arm, labelled scenarios only (M5)
	LatencyDelta LatencyDelta       `json:"latency_delta"` // (M6)
	WallMS       float64            `json:"wall_ms"`       // (M7)
	Outcomes     map[string]int     `json:"outcomes"`      // "better/slower" → scenarios (M8)
	Scenarios    []ScenarioOutcome  `json:"scenarios"`
	Negatives    Negatives          `json:"negatives"`
	Stability    Stability          `json:"stability"`
}

Summary is M1–M8 for one record (one model, one environment).

func Summarize 

func Summarize(reps []Rep, grades []Grade, labels Labels) Summary

Summarize computes every metric from observations and grades.

type Tokens 

type Tokens struct {
	ModelIn  int `json:"model_in,omitempty"`
	ModelOut int `json:"model_out,omitempty"`
	JevIn    int `json:"jev_in,omitempty"`
	JevOut   int `json:"jev_out,omitempty"`
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.