metronome

metronome

A protocol-agnostic Go load kernel: drive a unit of work at a controlled, live-adjustable rate across N workers, and measure latency and errors.

metronome knows nothing about HTTP, gRPC, Prometheus, configuration files or UI. You supply a Runner — one unit of work — and a RateController, and it gives you a stream of Results and an aggregator that turns them into percentiles.

go get github.com/RomanAgaltsev/metronome

Example

runner := metronome.RunnerFunc(func(ctx context.Context) metronome.Result {
	start := time.Now()
	resp, err := http.Get("http://localhost:8080/api/users/42")
	if err != nil {
		return metronome.Result{Start: start, Latency: time.Since(start), Err: err}
	}
	defer resp.Body.Close()
	n, _ := io.Copy(io.Discard, resp.Body)
	return metronome.Result{
		Start:   start,
		Latency: time.Since(start),
		Code:    strconv.Itoa(resp.StatusCode),
		Bytes:   n,
	}
})

d := metronome.Driver{
	Runner:      runner,
	Rate:        metronome.Ramp{Start: 10, End: 200, Over: 30 * time.Second},
	Workers:     16,
	MaxRequests: 5000,
}

stats := metronome.NewStats()
for r := range d.Run(context.Background()) {
	stats.Record(r)
}

fmt.Println(stats.Snapshot())
// 5000 req, 198.7 rps, 0.14% err (0 saturated), p50/p95/p99 12ms/41ms/88ms,
// corrected p95/p99 41ms/89ms, behind schedule 1.2ms

Snapshot is a plain struct — reach for the fields when you want them (snap.P95, snap.Codes, snap.Bytes). String() is there so every consumer does not reinvent the same summary line, and so the numbers that matter appear in the order they should be read.

You must drain the channel until it closes, or cancel the context — abandoning a live channel leaks the workers.

What's in it

Pacing model — read this before trusting the numbers

metronome offers two pacing modes and reports latency two ways. The defaults are conservative; the honest defaults for a latency-measuring tool are the other ones.

Closed loop (default)

Each worker waits for a rate-limiter token, calls your Runner, and only then asks for the next token. Simple, self-throttling, and the right choice when you are the backpressure mechanism (a control loop that deliberately probes for the point where the target sags). Two consequences:

• Rate sag. When the target slows, workers sit inside Do, tokens go unclaimed, and the achieved rate falls below the offered rate — silently.
• Coordinated omission. Latency is measured from when work actually started, so every stall suppresses exactly the samples that would have revealed it.

Open loop (Pacing: metronome.OpenLoop)

One dispatcher keeps the schedule and never blocks on the target. A unit that finds no free worker is delivered immediately as a Result whose Err matches errors.Is(err, metronome.ErrSaturated). Saturation therefore shows up as Snapshot.ErrorRate, not as invisible sag — and Snapshot.Saturated counts it separately, so "my generator ran out of workers" stays distinguishable from "the target failed". Workers becomes the maximum in-flight cap.

Open loop paces from one dispatcher goroutine, which sleeps once per unit of work. Below roughly a 250µs interval that sleep costs more than the interval, and the dispatcher — not the target — becomes the bottleneck: on the machine measured below, open loop holds 1,000 rps exactly and delivers only 74% of 5,000 rps, with zero ErrSaturated results, because the target was never asked. Snapshot.MaxScheduleLag is what makes that visible; see the table below. Closed loop does not have this ceiling (its Workers goroutines sleep concurrently, so each individual sleep is Workers times longer). Above ~1,000 rps, check MaxScheduleLag, and prefer closed loop if throughput matters more to you than schedule fidelity.

Raw vs corrected percentiles

Every Result from a Driver carries Scheduled, the time it should have been sent. Stats reports both:

• P50 / P95 / P99 — measured from when work started. Optimistic under stall.
• CorrectedP50 / CorrectedP95 / CorrectedP99 — measured from when the work was due, i.e. including the queueing delay a schedule-faithful client would have suffered.

Read them as a pair. A large gap means the generator queued: the raw numbers understate what a real client would experience by roughly that amount.

snap := stats.Snapshot()
fmt.Printf("p95 %v (corrected %v), achieved %.1f rps, %v behind schedule\n",
	snap.P95, snap.CorrectedP95, snap.RPS, snap.MaxScheduleLag)

The schedule is anchored: Scheduled is the run's origin plus one interval per unit dispatched, advanced whether or not the generator got there on time. A late generator is therefore measured against the schedule rather than redefining it. One worker offered 100 rps against a 50ms target achieves 19.9 rps and reports:

raw       P50=50.0ms  P99=50.0ms
corrected P50=412.9ms P99=816.1ms   MaxScheduleLag=765.7ms

The 816ms is the honest number: by the twentieth request, a client that kept to the 10ms schedule would have been waiting that long. (Before v0.3 both lines read 50ms.)

Who fell behind — you or the target?

Snapshot.Saturated and Snapshot.MaxScheduleLag answer that between them:

The third row is why MaxScheduleLag exists: ErrSaturated can only report a target that was too slow, never a dispatcher that was. At 5,000 rps open loop reports Saturated=0, MaxScheduleLag=330ms — nothing was wrong with the target.

Rolling window

Every number a Snapshot carries is cumulative over the whole run. That is right for the report you print at the end and wrong for anything watching a run while it happens: MaxScheduleLag is a running maximum, so one stall in the first second pins it for the rest of the run, and a target that stops answering never moves any cumulative number at all — there are no new Results to move them, so RPS goes on reporting the rate the run used to achieve.

RollingStats records into both views at once. Use Snapshot() for the end-of-run report — it means exactly what Stats.Snapshot() means — and Window() for a control loop, a progress line, or anything else reading a run live.

rs := metronome.NewRollingStats(metronome.Rolling{Window: 10 * time.Second, Buckets: 10})
for r := range d.Run(ctx) {
	rs.Record(r)
}

// live, from a control loop or a progress ticker:
fmt.Println(rs.Window())   // last 9.7s: 4821 req, 497.0 rps, 0.00% err (0 saturated), ...
// at the end:
fmt.Println(rs.Snapshot()) // 50210 req, 499.1 rps, 0.02% err (0 saturated), ...

The zero Rolling{} is valid: a 10-second window in 10 buckets on the wall clock over the NewStats histogram range. Pass a Clock to make window tests exact, the same ManualClock the Driver takes.

Window() divides by the time the ring actually covers, not by the nominal window, and reports that duration in Snapshot.Window — non-zero on a windowed Snapshot, zero on a lifetime one, and String() prefixes last 9.7s: so the two are never confused in a log. So three seconds into a ten-second window a healthy 100 rps reads as 100.0 over Window: 3s rather than 30.0 over a window two-thirds empty, and at steady state the figure sawtooths across [window − interval, window] because the newest bucket is always partial. The numbers are correct for the period they cover; that field is what the period is.

Rotation happens on read as well as write, which is the whole point: with no Record traffic to drive it, only a read can notice that time has passed, so a full stall drains the window to zero instead of freezing it at the last healthy figures.

Measured on the machine described under Measured accuracy:

Record is the hot path and pays a flat 55 ns. Window() is not: it merges the live buckets into a dense scratch target and reads the percentiles off that, so it costs roughly live × countsLen — 0.7 ms at the default range and a full ring. Poll it at 1–10 Hz from a control loop, not per request. Empty buckets are skipped, so the stall case a control loop polls hardest in is the cheap one.

v0.9's sparse buckets did not make this cheaper, and the honest reason is worth stating: the cost is in the destination, not the sources. A sparse bucket merges as a walk over its entries rather than over a 17,408-slot array, but the scratch target is reset and quantile-read across its whole array either way, and that dominates. The one claim that did change is the old "independent of how many Results are in them": a sparse bucket's merge now scales with the distinct latencies it holds, so a busier bucket costs marginally more until it promotes. Reproduce with:

go test -run '^$' -bench 'BenchmarkStatsRecord|BenchmarkRollingStats' -benchtime=2s -benchmem ./...

Memory

A RollingStats is sized for Buckets+2 histogram pairs — the ring, the lifetime aggregate and a scratch merge target — so Buckets multiplies memory as well as resolution. Price any configuration before building it with Rolling.Bytes().

These are ceilings, not what a run costs. Since v0.9 a ring bucket starts as a sparse value-to-count store and allocates a dense HDR histogram only once it holds enough distinct latencies to be worth one — about 2,785 distinct microseconds at the default range. Rolling.Bytes() is the budget above; RollingStats.Bytes() reports what is actually held, and is typically far below it. Measured on the 266 MiB row:

A bucket that does promote keeps its histogram for the life of the ring, so a run whose latencies genuinely spread wide converges on the ceiling — which is why the ceiling is still the number to budget with. The last two table rows remain the rule: narrow the histogram range when you want a large ring. Rolling.Bytes() panics on exactly the configurations NewRollingStats panics on, so pricing an unbuildable config reports the problem rather than a number for it.

The representation is invisible in output: every Window() and Snapshot() field is identical whether a bucket is sparse or dense, clamp counters included, and a property test compares a sparse ring against a dense one on random streams to keep it that way.

Per-endpoint breakdown

A Mix of ten endpoints reports one P99 that no endpoint exhibits. LabeledStats splits the stream on one Result.Labels key:

stats := metronome.NewLabeledStats(metronome.Labeled[*metronome.Stats]{
	Key: "endpoint",
	New: metronome.NewStats,
})
for r := range d.Run(ctx) {
	stats.Record(r)
}

stats.Snapshot()                      // the total — same as a plain Stats
stats.Series()["search"].Snapshot()   // just that endpoint

The total is exactly what a plain Stats fed the same stream would have produced, so existing code holding a *Stats can hold one of these instead and see its numbers unchanged. The breakdown is something it gains, not something it trades for. Every Result lands in the total and in exactly one series, so Σ Series().Count == Snapshot().Count — the breakdown reads as a decomposition rather than as a sample.

LabeledStats is generic over the child recorder, so use Labeled[*metronome.RollingStats] for a trailing window per endpoint — stats.Series()["search"].Window(), typed, no assertion. That is the one that answers which endpoint is slow right now, which is the live question.

A Result whose Labels is nil, lacks the key, or holds an empty string for it goes to a named __none__ series rather than being dropped, because "which requests were not labelled?" is exactly the debugging question.

Cardinality is memory. Each series is a whole aggregate, not three counters, so MaxSeries (default 100) caps it and further values share an __overflow__ series:

The bottom-right figure is the one to know before you type Labeled{Key: "endpoint"}. It is not a defect — it is ten buckets of two histograms for each of a hundred endpoints, every term of which you asked for — but series are created lazily, so a run only discovers it if the cardinality turns out to be real. That is what the cap is for: a label value that turns out to carry a request ID rather than a route name costs you reporting detail, not memory.

stats.Bytes() reports what is currently held, and -1 if the child type does not report its own size (which a custom Recorder is not required to do). The __none__ series counts against MaxSeries; __overflow__ does not.

Measured cost of the breakdown, on the machine described under Measured accuracy. Every row records the identical Result, so the delta is the breakdown and nothing else:

A breakdown costs about 3.3× a flat Stats — roughly +250 ns per Result, zero allocations. Two aggregates rather than one is about half of that; the rest is the label lookup, the RWMutex and the extra lock traffic. It is not free, and at 350 ns it is still far below any real request.

The shape matters more than the level: 100 series costs 0.90× what 1 series costs, so the map does not bind as cardinality grows. With a single series every goroutine contends on that one child's mutex as well as the total's; spreading the traffic across a hundred children removes the child contention. A breakdown gets cheaper as it gets wider.

Stamping the label is a separate cost, paid by whoever sets Result.Labels whether or not anything reads it, and it is not counted above. Recording a Result carrying a one-entry Labels map into a plain Stats — which ignores it — costs 354 ns against 91 ns with Labels nil: +263 ns and 2 allocations, for the map alone. Leave it nil when you are not breaking down. Reproduce with:

go test -run '^$' -bench 'BenchmarkLabeledRecord|BenchmarkLabelStampCost' -benchtime=2s -benchmem ./...

Warmup exclusion and fan-out

The first seconds of a run measure cold connection pools, TLS handshakes that will be reused and a target that has not warmed up — none of which is the system under test. After keeps them out of the measurement without changing the load:

report  := metronome.NewStats()
byRoute := metronome.NewLabeledStats(metronome.Labeled[*metronome.Stats]{
	Key: "endpoint", New: metronome.NewStats,
})

measured := metronome.After(10*time.Second, metronome.Multi(report, byRoute))
metronome.Drain(d.Run(ctx), measured)

fmt.Println(report.Snapshot())
fmt.Printf("measured %d, excluded %d as warmup\n",
	report.Snapshot().Count, measured.Skipped())

Count + Skipped() is always the whole population, so an exclusion is auditable. "Measured 4,500 of 5,000 — 500 excluded as warmup" is the honest line; "4,500 requests" alone invites the reader to wonder where the rest went, and a threshold evaluated over a silently reduced population is the thing this exists to prevent.

The boundary anchors on the first Result's Scheduled stamp, so it is fixed by the stream rather than by when you built the recorder — measured against the schedule, not against when the generator got there. Pair it with Phased to keep one source of truth: metronome.After(rate.PhaseEnd(0), stats) cannot drift from the phase table the way a repeated literal duration can. Use AfterTime when the boundary must be exact (record clock.Now() immediately before Run and pass origin.Add(warmup)), and AfterN to count units instead of time. After(0) admits everything and never anchors, so an unset --warmup means exactly nothing.

Order is visible in the expression, and both readings are valid: After(w, Multi(report, byRoute)) gives two views of one warmed population; Multi(After(w, report), byRoute) gives a warmed report beside an unwarmed breakdown.

Multi is synchronous. Everything in one should be an in-memory aggregate. Time spent in Record is time not spent receiving from the result channel, so a slow writer fills the channel, then the Driver's delivery backlog, and in open loop surfaces as ErrSaturated — which reads as "the generator ran out of workers" and is counted inside Errors and ErrorRate. A slow file writer in a Multi can therefore present itself as a target problem. Put I/O behind your own buffered goroutine, where you own the policy for what happens when it falls behind.

Every recorder in a Multi receives the same Result, and Result holds Labels by reference, so treat Results as read-only: a recorder that mutates the map corrupts what the others see, including the series LabeledStats picks.

Composing a load shape

A RateController is a pure function of elapsed time, so shapes compose by arithmetic. Sine is the one curve composition cannot reach; Sum, Repeat and Scale reach the rest.

// Diurnal: a smooth curve between 100 and 1,000 rps, once an hour, starting low.
rate := metronome.Sine{Min: 100, Max: 1000, Period: time.Hour}

// A spike on a baseline: 100 rps, rising to 500 for one minute in every ten.
rate = metronome.Sum(metronome.Constant(100),
	metronome.Repeat(metronome.Phased{Phases: []metronome.Phase{
		{Duration: 9 * time.Minute, TargetRPS: 0},
		{Duration: 1 * time.Minute, TargetRPS: 400},
	}}, 10*time.Minute))

// The same agreed profile at half intensity — a canary, or a smaller blast radius.
rate = metronome.Scale(0.5, rate)

There is no Burst controller, and that is the point. A burst is a constant plus a repeating phase table, and the same three combinators also give you a burst on a ramp, a burst on a sine, two bursts at different periods, and a burst that is itself a curve. (The Burst in the table above is unrelated: it is the rate limiter's bucket size.)

Sine starts at Min and peaks at Period/2, so a run opens at its lowest load and rises, the way Ramp opens at Start. It is already periodic, so it is never wrapped in Repeat — Repeat is for finite shapes such as Phased and Ramp. Pair a phase table with its own length, Repeat(p, p.Duration()), and the cycle cannot drift from the table.

Two more compositions worth knowing, neither of which required Adaptive to change: Sum(metronome.Constant(50), adaptive) puts a floor under a control loop, so a feedback signal that collapses cannot drive the run down to nothing; and Scale(0.5, profile) reruns a named profile at half intensity.

Sum, Repeat and Scale panic at construction on a nil controller — Sum naming the argument index — rather than mid-run from the rate-updater goroutine. A non-positive Repeat period delegates to the inner controller instead of cycling, and a non-positive Sine.Period reports Min.

⚠️ The controller is sampled ten times a second — not once per request. The Driver re-reads it every 100 ms and the limiter holds the last value in between. So shape features shorter than about 200 ms do not exist: a one-second Sine gets ten samples and renders as visible steps, and a Repeat period near the sampling interval aliases into a shape unrelated to the one you asked for. Keep Period, and any spike inside it, at a second or more — ten times the sampling interval, which is where the rendered shape stops depending on it. The upside of the same fact: math.Cos runs 10 times a second for the whole run, so a composed controller costs nothing measurable no matter how deep it is.

Measured accuracy

Measured on an AMD Ryzen 5 3600 (6 cores / 12 threads), Windows 11, Go 1.26.6, machine otherwise idle. Adherence is achieved rps ÷ offered rps; 1.00 is perfect.

The 0.74 is real and reproducible, and it is a limit of the generator, not of the target — see the open-loop note above. That run reports Saturated=0 and MaxScheduleLag=330ms, which is how you would tell without reading this table. Anchoring the schedule in v0.3 made the shortfall visible, not smaller: the adherence numbers are unchanged from v0.2. Reproduce with:

go test -run '^$' -bench BenchmarkDriverPaced -benchtime 3x ./...

Per-request kernel overhead (no-op Runner, unlimited rate, Workers = GOMAXPROCS): 600 ns/op closed-loop and 895 ns/op open-loop, i.e. a plumbing ceiling near 1.7M rps and 1.1M rps respectively. Note the gap between that ceiling and the 5,000 rps adherence figure above: metronome's limit at realistic rates is sleep granularity, not CPU. Stats.Record costs 201 ns/op under full contention (RollingStats.Record 256 — see Rolling window; LabeledStats under Per-endpoint breakdown). Reproduce with:

go test -run '^$' -bench 'BenchmarkDriverOverhead|BenchmarkStatsRecord' ./...

Status

v0.9 — API is stable in shape and pinned by two consumers, but pre-v1: minor versions may carry small breaking changes, always with a migration note in the CHANGELOG. Pin an exact version.

Nothing has been removed or changed in meaning since v0.3; v0.4 through v0.9 are additive but for one named exception. v0.9 changed what RollingStats.Bytes() reports — the memory actually held rather than the budgeted ceiling. No signature changed; Rolling.Bytes() is still the ceiling. The Driver, Result and Snapshot a v0.4 consumer compiled against behave identically today.

Used by

• crescendo — PromQL-driven adaptive load: a Mix of endpoint Runners under an Adaptive controller steered by a Prometheus feedback loop.
• quiver — qv load: one Runner over a saved API request, Constant/Ramp, MaxRequests, Stats summary.

License

MIT