README
¶
sample
A row of logits in, a token out.
The chain is llama.cpp's default one, in its order:
- top-k keeps the
khighest logits. With a vocabulary of 262144 and akof 64, sorting the row would cost more than the draw it serves, so the selection runs through a heap ofkentries — the root is the worst kept candidate, and anything that does not beat it is dropped on sight. Only the survivors are sorted. - top-p softmaxes the survivors and keeps the shortest prefix whose
probabilities reach
p, including the token that crosses the threshold. One candidate always survives: an empty distribution has nothing to draw from. - temperature divides what is left before the final softmax. Zero or less
never reaches this point —
Pickreturns the highest logit straight away. - the draw walks the cumulative probabilities against one float from the generator.
Ties, everywhere, go to the lower identifier, which is what llama.cpp does and what makes a greedy run reproducible.
The generator is math/rand/v2's PCG, seeded from Params.Seed. A sampler owns
it, so two samplers in the same process do not disturb each other and a seed
plus a sequence of logit rows determines the tokens exactly.
s := sample.New(sample.Params{Temperature: 1, TopK: 64, TopP: 0.95, Seed: 7,
PenaltyLastN: 64, PenaltyRepeat: 1.1})
s.Seed(prompt) // the penalties read the conversation, not only the answer
s.Constrain(g) // optional: a grammar the draw has to stay inside
id := s.Pick(logits)
Defaults() returns the values Gemma 4's file declares for itself; gemma
reads that file's own general.sampling.* keys and falls back to them.
Two things sit around that chain.
The penalties. PenaltyRepeat, PenaltyFreq and PenaltyPresent over the
last PenaltyLastN identifiers, applied before top-k, with llama.cpp's own
arithmetic — a logit above zero is divided, one at or below it is multiplied,
which is llama.cpp's fix for a division that would make an unlikely token
likelier. Seed puts the prompt into that window, as llama-server does before
its first draw, and Pick puts whatever it draws there.
The window is what makes the shortcut exact. A penalty can only lower a logit, so the k best penalised candidates are among the k+u best raw ones, u being the identifiers in the window: the chain takes k+u through the heap, penalises those, and cuts back to k without ever walking the row. Weights that could raise a logit — a repeat below one, a negative frequency or presence — break that argument, and the sampler notices and penalises a copy of the row instead.
The constraint. Constrain hands the sampler something that says which
tokens may come next; grammar.Grammar is what that is in practice. The chain
draws first and asks about the one token it drew, and only a refusal costs
anything: then it filters a prefix of the sorted row, widening it only while
too few candidates survive, and tests a token's lead byte against the grammar's
own table before asking the grammar anything.
What is still not here: min-p, typical-p, locally typical sampling, mirostat, DRY. They are additions to this chain rather than changes to it, and nothing in golem asks for them yet.
Documentation
¶
Overview ¶
Package sample turns a row of logits into a token.
The chain is the one llama.cpp runs by default, in its order: top-k cuts the row down to the most likely few, top-p cuts it again at a share of the mass, temperature reshapes what is left, and a draw from the resulting distribution names the token. A temperature of zero short-circuits all of it and takes the highest logit.
Nothing here knows what model produced the row.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Constraint ¶ added in v0.27.0
type Constraint interface {
// Allows reports whether this token may come next, changing nothing.
Allows(id int32) bool
// Accept advances over a token that was drawn.
Accept(id int32)
// FirstBytes is the lead bytes an allowed token may start with, as a
// superset: a sweep tests one of these before it asks Allows anything.
FirstBytes() *[256]bool
// FirstByte is a token's own lead byte, or zero for a token whose piece
// says nothing about whether it is allowed.
FirstByte(id int32) byte
}
Constraint is the part of a grammar the sampler drives.
type Params ¶
type Params struct {
// Temperature divides the logits before the draw. Zero or less is greedy.
Temperature float32
// TopK keeps that many candidates. Zero or less keeps all of them.
TopK int
// TopP keeps the shortest prefix of the sorted candidates whose
// probabilities reach it. One or more keeps all of them.
TopP float32
// MinP keeps the candidates at least that share as likely as the best
// one. Zero or less keeps all of them.
MinP float32
// Seed fixes the run: the same seed over the same logits draws the same
// tokens.
Seed uint64
// PenaltyLastN is how many of the conversation's last tokens the three
// penalties below read. Zero turns them off; a negative one reads the
// whole conversation.
PenaltyLastN int
// PenaltyRepeat divides the logit of a token in that window, or
// multiplies it when the logit is negative. One is no penalty.
PenaltyRepeat float32
// PenaltyFreq is subtracted once per appearance in the window,
// PenaltyPresent once for any appearance at all. Zero is no penalty.
PenaltyFreq float32
PenaltyPresent float32
// Constraint is what the draw has to stay inside — a grammar, in
// practice. It belongs to one answer rather than to a file, and a
// sampler that outlives an answer takes Constrain instead.
Constraint Constraint
}
func Defaults ¶
func Defaults() Params
Defaults are the values Gemma 4's own file declares under general.sampling, with llama.cpp's own defaults for what it does not declare: the penalties, and a min_p of 0.05. The file's top-k and top-p leave a tail that a temperature of one still reaches, and a 12B writing French drew broken words out of it a few times in a hundred answers without it.
type Sampler ¶
type Sampler struct {
// contains filtered or unexported fields
}
func (*Sampler) Accept ¶ added in v0.27.0
Accept puts a token into the window. Pick calls it with whatever it draws, so a caller that hands Pick to a speculative step has nothing else to do.
func (*Sampler) Constrain ¶ added in v0.27.0
func (s *Sampler) Constrain(c Constraint)
Constrain hands the sampler a constraint to draw inside. Every token Pick names from here on is one the constraint allowed, and is accepted into it.
func (*Sampler) Pick ¶
Pick names a token. The logits are read, never written.
Whatever it names joins the penalty window, so a caller that hands Pick to a speculative step has nothing else to keep up to date.
func (*Sampler) Seed ¶ added in v0.27.0
Seed puts a whole run of tokens into the window at once, which is what the prompt is: llama-server feeds every prompt token to the sampler before the first draw (tools/server/server-context.cpp:254-260), so a repeat penalty sees the conversation and not only the answer.
Source Files