golem-server

golem-server

An OpenAI-compatible API over a GGUF, on the CPU or — with -vulkan — with the model's blocks and logit head on a Vulkan device. Gemma 4, Qwen3 or Qwen3.8: the file declares its own architecture and engine/ opens whichever implements it, so this command names none of them, and tools work on all three.

go build ./cmd/golem-server
./golem-server -model gemma-4-E2B-it-QAT-Q4_0.gguf -addr 127.0.0.1:8080
./golem-server -model Qwen3-4B-Q4_0.gguf -addr 127.0.0.1:8080

One model per server, and the request's model field routes nothing: there is one set of weights, so there is nothing to route to. -model names the file; /v1/models reports it.

The endpoints:

Tools

Declare them the way the API declares them, and the calls come back the way it returns them:

curl -s localhost:8080/v1/chat/completions -H 'content-type: application/json' -d '{
  "messages": [{"role": "user", "content": "What is the weather in Lyon?"}],
  "tools": [{"type": "function", "function": {
    "name": "get_weather", "description": "Current weather in a city.",
    "parameters": {"type": "object",
      "properties": {"city": {"type": "string", "description": "The city."}},
      "required": ["city"]}}}]}'

{"choices":[{"index":0,"message":{"role":"assistant","content":"",
  "tool_calls":[{"id":"call_1","type":"function",
    "function":{"name":"get_weather","arguments":"{\"city\":\"Lyon\"}"}}]},
  "finish_reason":"tool_calls"}]}

Running the function is the client's part. Send the result back as a tool message carrying the call's tool_call_id, as the protocol has it, and the model speaks on inside the same turn — that is the template's own rule, not a choice made here.

Streamed, a call never arrives in fragments. Prose leaves as it is drawn; from the moment <|tool_call> appears the output is held back, and the whole call leaves in a single tool_calls delta once it has closed. OpenAI's API allows fragments and clients do reassemble them, but half a function's arguments is a thing nothing can check.

Reasoning

What the model thinks before it answers leaves in reasoning_content, where llama-server puts it, and never in content: Gemma's thought channel (<|channel>thought … <channel|>) and Qwen's <think> … </think> alike. Streamed, it arrives as reasoning_content deltas while it is drawn. The markers themselves never leave, nor do the newlines the template writes around them. A thought left open when the answer ends is reasoning to the end, as the template reads it back.

The server never asks for thinking, and Gemma's template closes an empty thought channel at the end of the prompt. The 12B sometimes opens one anyway; it is dropped here rather than sent to the client as prose.

Several conversations, in one pass

-parallel is how many conversations the server holds at once, and they are answered together rather than in turn. Each slot is a KV cache holding its own conversation; the context is cut rather than multiplied, as llama.cpp's --parallel cuts it, so -context 4096 -parallel 2 is two conversations of 2048 positions and the memory is what it was.

What makes them cheap together is that they go through the model together. One goroutine owns the weights, and whatever is waiting when it builds a pass goes into that pass: four conversations each wanting one token are four tokens in one read of a gigabyte of weights, rather than four reads. That is llama.cpp's continuous batching, and on this engine it is worth this much — Gemma 4 E2B on an eight-thread i7-9700K, clients asking for 64 tokens each at once:

Against llama.cpp's own server on the same machine and the same file (llama-server -dev none -ngl 0 -t 8 -c 4096 -np 4 -cb), three rounds, a different prompt every round so that neither server is answering out of a cache it filled a moment ago:

Where the four-client deficit is, exactly: GOLEM_DEBUG_BATCHES=1 over that run says the batches are as wide as they can be — 63 of the 67 passes carried all four conversations, 3.88 on average — and that a pass with its head takes 60ms. What is left is the 655ms spent reading the four prompts, which is not scheduling but the rate this engine reads prompts at. Generation is at parity; the prompt is the half that is behind, which gemma/README.md already says.

The engine's own figure for generation, without a server around it, is TestMixedBatchCost in gemma/: 21.4 tokens a second for one conversation, 42.3 for two, 73.3 for four, 104.1 for eight.

Two things bound it. Generation is limited by reading the weights, so the first conversations added are nearly free and the later ones are not — by eight, the arithmetic is what is left. And the output head is the largest matrix in the model: it is read once for every state being scored in a pass, which is why the scoring travels with the pass rather than after it.

On the card

-vulkan -parallel N batches the same way. The caches are cut the same way too: one ring a conversation, laid end to end in the buffer a block already had, with the slot travelling beside the position in vk/attention.go's position buffer — so the card holds N conversations for the memory one of N times the context held. TestMixedBatchCostVulkan in gemma/, the 26B A4B on an RX 9070 XT:

Taken from four hundred and forty-eight positions of context, because a token drawn at position 64 costs 7.6ms on this model and one drawn at 512 costs 14.1ms — the attention reads everything before it, and a table taken at the shallow end would say a number no server ever sees.

That is 1.53, 2.57 and 3.11 of one conversation. What keeps it from being two, four and eight is that generation is limited by reading the weights only while there is nothing else to do: by eight columns the arithmetic and the head are what is left.

A mixture had a threshold in the way of this. It answers a single column by reading the eight matrices that column routed to, and a prompt by reading each expert once for the columns that chose it — and it used to switch to the second at two columns, so a token drawn for each of two conversations read a hundred and twenty-eight experts to answer sixteen pairs. The two costs cross near thirty-two on this checkpoint, which is where vk/mixture.go's byExpertFrom now sits; the by-column kernels take a column offset so they can answer a handful of columns rather than one.

The scores kernel is what limits how a pass may be mixed. It answers thirty-two columns to a workgroup off the keys of the union of their ranges, and two conversations cannot share that scratch — a position means a different entry of the cache in each. So a pass is cut into runs of one conversation before the tiles are laid out. A prompt is one run and tiles as it always did; a token drawn for each of four conversations is four runs of one column, which is the shape generation has anyway.

How wide a pass may be is the other thing the card changes. A prompt is read in chunks of thirty-two positions on the processor, because past sixty-four the activations stop fitting in its caches; a card is idle at that width and reads a prompt some five times faster at two hundred and fifty-six. Positions a second by the width, the same 26B on the same card:

Runner.PassWidth is which of the two this server uses, and main.go asks the model where its blocks are and says so once. Five hundred and twelve is the widest the stack carries and also the longest pass — 93ms, against 51ms at 256 — so a conversation waiting on its next token waits that much longer for one that is reading a prompt. That wait is a fraction of a prompt, once; the rate is every prompt.

End to end, through the API, with prompts of about five hundred and forty tokens and nothing else on the card:

The prompt figure was 804/s before the width followed the device, and two clients drew 100/s together before the mixture's threshold moved.

Qwen3.8's pipeline holds its slots the same way, with one difference: a delta net keeps a state matrix a head rather than a ring, so a slot's share of the buffer is a whole copy of every recurrence. The slot travels beside the position all the same, and a pass carries several conversations, a run of columns each; only the recurrences and the scores are dispatched a run at a time. Bonsai 2 27B, one token for each conversation drawing:

Five clients drawing two hundred tokens each went from 36.5 tokens a second together, when every conversation was a pass of its own, to 79.6. Past four the ternary mat-vec stops being a reading of the weights and becomes arithmetic, which is where the next gain is.

Waiting for a pass

The runner waits a fraction of a pass — an eighth of what the last one took, at most two milliseconds — for the other conversations in flight to arrive before going without them. A lone client cannot feel that; a second client is worth far more than it costs.

GOLEM_DEBUG_BATCHES=1 says what went into each pass, which is how the table above was checked rather than assumed.

Which slot answers is decided as llama.cpp's server decides it. The free slot whose tokens share the longest prefix with this prompt wins, provided the shared part is a tenth of the prompt or more; failing that, a slot holding no conversation; failing that, the one nobody has come back to in longest. Three requests measured on a Qwen3 0.6B — Alice's conversation, Bob's, then Alice's next turn:

A request that waited for a slot — all of them held — says so in its log line, next to the slot that answered it:

POST /v1/chat/completions 200 284 bytes in 139ms — slot 0 — 21 prompt in 37ms (566.4/s), 8 drawn in 102ms (78.65/s), length

A client that hangs up while queued takes its request with it, and never reaches a slot.

The cache across stateless requests

/v1/chat/completions carries the whole conversation every time; the cache does not want to be rebuilt every time. So the server remembers which tokens sit in the cache, compares them with the prompt it just rendered, and prefills only the divergence. A conversation growing by one exchange costs one exchange.

Rewinding is where it gets particular. A sliding-window block keeps its keys in a ring of C slots, the window and one pass more, position p in slot p mod C. Writing up to position Q and then going back to P has overwritten the slots of positions up to Q-C, and some of them can still be visible from P. The server remembers which position each slot last received and resumes at the latest position whose window is intact, feeding everything after it again. A rewind that wrapped nothing feeds only the divergence, and appending, the common case, rewinds nothing.

A recurrent model cannot rewind at all: a delta net's state has read every position it holds, and there is no taking one back out. A prompt that parts from what a slot holds used to be read again from nothing. So a slot keeps a few copies of its state, -checkpoints of them, taken where a prompt parted from the one before it and every thousand and twenty-four positions of a long one, and a prompt that parts starts from the latest copy it shares. The attention's keys below that position are still the conversation's own; the copy is all it takes. A copy is a hundred and fifty-one megabytes on the 27B, and the card grants fewer than asked when they would push the head off it.

What it is worth is what a village is: five characters, each asked a prompt that starts with the same long sheet and then says what is happening now, which is new every time. Fifteen answers on Bonsai 2 27B took 196 seconds and 44 with the copies, the median wait from 61 seconds to 8.6: after its first turn a character's prompt is read from where its sheet ends, four hundred positions where it was three thousand eight hundred.

-cache-ttl bounds how long a conversation's tokens stay in memory. Say plainly what it does not do: the cache is allocated once at startup and this frees none of it. What it does is stop the server from holding one client's conversation indefinitely, and give the next request a clean state. 0, the default, never forgets.

A client that hangs up

Cancelling a request stops the drawing. With one model behind one lock, an answer nobody is waiting for is not merely wasted: it is the next request's wait. The generation loop watches the request's context and returns as soon as the connection is gone.

Where the time goes

Every request leaves a line on standard error, split between the two halves of the wait — they have nothing in common and nothing to gain from being added up:

POST /v1/chat/completions 200 268 bytes in 51.31s — 3919 prompt in 51.226s (76.5/s), 2 drawn in 70ms (28.70/s), stop

A first call carrying a large system prompt and a page of tool declarations is almost entirely prefill, and prefill is the slower half of this engine. The cache is what pays it back: the next turn of that conversation shares the whole prefix and reads only what was added. A client that rewrites the head of its conversation between turns — an hour stamped into the system message, tools reordered — diverges at position zero and repays the whole prompt every time. The prompt figure in the log says which of the two is happening.

Sixteen, sixty-four, and up to five hundred and twelve positions per batch were measured against thirty-two on this engine, and thirty-two won every time: at 3919 positions, 76.2/s against 71.7 at sixty-four and 46.4 at five hundred and twelve. Past that the activations stop fitting in the caches.

An answer a schema can read

response_format in its two shapes, and llama-server's grammar field beside them:

curl -s localhost:8080/v1/chat/completions -d '{
  "messages": [{"role": "user", "content": "Lyon, please."}],
  "response_format": {"type": "json_schema", "json_schema": {"name": "city", "schema": {
    "type": "object",
    "properties": {"city": {"type": "string"}, "population": {"type": "integer"}},
    "required": ["city", "population"],
    "additionalProperties": false}}}}'

{"type": "json_object"} asks for any JSON value at all; "grammar": "root ::= …" asks for whatever the GBNF describes. A request carries one of them, never two.

The token that would break the document is refused before the draw, and the turn cannot end while the braces are open. A schema that does not compile, a grammar that does not parse and a request carrying both are refused with a 400 before a slot is taken: a request that will not be answered has no business waiting behind one that will.

The vocabulary a grammar reads through is decoded on the first request that asks for one and shared by every request after. A server nobody asks a schema of never builds it.

The four penalty fields — repeat_penalty, repeat_last_n, frequency_penalty, presence_penalty — are llama.cpp's, and the window is seeded with the prompt before the first draw, as llama-server does.

A model that reasons answers without it unless the request asks, llama-server's way: "chat_template_kwargs": {"enable_thinking": true}. The reasoning then streams apart, as reasoning_content.

What it refuses

With a 400 and OpenAI's error envelope, rather than answering something else: n above 1, logprobs, tool_choice beyond auto and none, a conversation with no message, a tool result answering no call, and a prompt longer than -context. In a schema: pattern, a numeric bound, and a $ref into another document — each by name, because a constraint silently dropped is worse than a request refused.

Not implemented at all: /v1/completions, embeddings.

Pictures and recordings are served when the model was opened with -mmproj, and refused by name when it was not. A turn's content may be an array of parts: an image_url carrying a data: URI or a path on the server's own filesystem, and an input_audio carrying bare base64 or such a path. An http URL is refused on purpose — a server that fetches what a prompt names is a server that can be aimed.

Speech to text

When started with -stt <dir> (or GOLEM_STT), golem-server serves POST /v1/audio/transcriptions using the Kyutai STT model:

# Standard transcription:
curl -s http://localhost:8080/v1/audio/transcriptions \
  -F "file=@clip.wav" \
  -F "model=stt"

# Streaming transcription:
curl -s http://localhost:8080/v1/audio/transcriptions \
  -F "file=@clip.wav" \
  -F "model=stt" \
  -F "stream=true"

Embeddings

With -embed <file> (or GOLEM_EMBED), a nomic-embed-text-v2-moe GGUF — the one ollama pulls as nomic-embed-text-v2-moe — answers three endpoints. It may sit beside a conversation model and an STT, or alone:

./golem-server -embed nomic-embed-text-v2-moe.f16.gguf -addr 127.0.0.1:11434

curl -s localhost:11434/v1/embeddings -d '{"input": ["search_query: golem", "search_document: a Go engine"]}'
curl -s localhost:11434/api/embed     -d '{"model": "nomic-embed-text-v2-moe", "input": "search_query: golem"}'

Each keeps the manners of the API it copies, because clients depend on them:

• /v1/embeddings answers unit vectors, as floats or — what OpenAI's own client asks for unless told otherwise — as base64 of little-endian float32. A text past the model's 512 positions is refused, as OpenAI and llama-server refuse it.
• /api/embed answers unit vectors and cuts a text that is too long, unless the request says "truncate": false.
• /api/embeddings, ollama's older endpoint, answers one vector, not normalized, as ollama does.

dimensions, on the first two, keeps a prefix of the vector and renormalizes it: the checkpoint is trained so that a prefix is a vector.

A request's texts go through the model in one pass, and so do the texts of requests that arrive while a pass is running: they wait for it, and are carried together by the next. A pass that would carry only a few sentences waits up to three milliseconds for more, because its cost is mostly fixed. With -vulkan the embedder's blocks go on the card, beside whatever else is there. nomic/README.md has the speed against llama.cpp and ollama, on the processor and on the card, and the vectors against both.

Pictures

With -krea2 <dir> (or GOLEM_KREA2), a ComfyUI directory holding Krea 2's three files, it draws pictures at /v1/images/generations, alone or beside the rest. krea2/README.md says what it draws and how close to ComfyUI.

./golem-server -krea2 /mnt/data/dev/ComfyUI -krea2-keep -addr 127.0.0.1:8083

curl -s localhost:8083/v1/images/generations \
  -d '{"prompt": "a red fox in the snow, photo", "size": "768x1024", "seed": 5}'

It is OpenAI's endpoint with what the ComfyUI mobile front sends beside it: negative_prompt, steps (8), cfg (1, which never reads the negative prompt) and seed (drawn when absent or -1). The answer is one picture as b64_json, a PNG, and the seed that drew it. n above 1 and response_format: url are refused. One picture is drawn at a time.

-krea2-keep leaves the twelve-gigabyte DiT on the card between pictures; without it each picture reads it again, which is most of its time when the file is not in the page cache.

Decisions

With -laya <dir> (or GOLEM_LAYA), a Laya checkpoint directory as Hugging Face ships it, it answers POST /v1/systemone, the endpoint TypeSafe Jev and Laya's own laya-serve answer on, with the same request and the same answer. laya/README.md says what the model is and how close to PyTorch.

./golem-server -laya convaiinnovations/laya -vulkan -addr 127.0.0.1:8000

curl -s localhost:8000/v1/systemone -d '{
  "state": {"document": "I was charged twice. Please fix this ASAP."},
  "questions": {"billing": {"type": "noul", "instructions": "Is this ticket about billing?"}}
}'

Refusals are written as laya-serve writes them, {"detail": "…"}: 400 for a body that is not the request, 413 past a mebibyte or sixty-four questions, 422 for a question Laya cannot read. Unknown fields are ignored, model among them: the server carries one checkpoint. Requests take turns on the model, and on an RX 9070 XT a request of three questions and 225 positions is answered in 14 ms over HTTP once the first has been.

Flags

Sampling follows the file's own values unless the request names temperature, top_p, top_k or seed.