golem

Gemma, Qwen and Pocket TTS in a single static Go binary.
No Python. No cgo. No GPU required — but with -vulkan it matches or exceeds llama.cpp's Vulkan performance on AMD hardware.

A golem is inert matter given a voice. That is what these engines do to a file of weights.

Golem is a set of inference engines written in pure Go. Run Gemma 4, Qwen3, Qwen3.8 and Kyutai Pocket TTS locally with go build, a GGUF file, and your CPU — or your Vulkan GPU.

✨ Features

• Zero Friction: Compiles to a single static binary. No Python environment, no cgo, no runtime to install.
• Pure Go, four dependencies: purego for the Vulkan loader, and three file formats the standard library does not read — WebP, MP3, FLAC. CGO_ENABLED=0 go build ./... passes.
• OpenAI Compatible: Drop-in replacement for OpenAI API clients, tool calls included.
• Multimodal: Text, Vision (images) and Audio (WAV/MP3/FLAC) via Gemma 4.
• Verified, not asserted: no layer is deemed correct until its intermediate activations match llama.cpp or PyTorch, waypoint by waypoint.
• Fast on CPU: keeps pace with llama.cpp on tuned AVX2 kernels — ahead reading prompts, level generating, except on the smallest model, where the weights stop being the cost and it says so.
• Vulkan GPU: bound through purego rather than cgo. Measured on AMD against llama.cpp's own Vulkan build: ahead of it reading prompts on all five models, and generating on both Gemma ones. The table below says where it is behind, and by how much.
• Serves several clients at once: -parallel N holds N conversations and carries a token for each of them through one read of the weights, on the card as well as on the processor — Qwen3.8 on the card excepted, for a reason written below.

🚀 Quickstart

Download a GGUF and run. No configuration file — the engine reads general.architecture out of the file and opens whichever engine implements it, so no command here names a model family.

CLI (chat in the terminal)

go build ./cmd/golem-cli

./golem-cli -model gemma-4-E2B-it-QAT-Q4_0.gguf -p "Explain a mutex in one sentence." -stats

Server (OpenAI-compatible API)

go build ./cmd/golem-server

./golem-server -model Qwen3-4B-Q4_0.gguf -addr 127.0.0.1:8080

-parallel N cuts the context into N slots, each holding its own conversation.

👥 Several conversations, one pass

Whatever is waiting at the moment a pass is built goes into that pass. Four clients each wanting a token are four tokens in one read of a gigabyte of weights, rather than four reads — llama.cpp's continuous batching, and it now works on the card as well as on the processor. The context is cut between the slots rather than multiplied, so -context 4096 -parallel 4 is four conversations of 1024 and the memory is what it was.

Gemma 4 26B A4B on an RX 9070 XT, drawing a token for each conversation from 448 positions of context:

Through the API, with prompts of about five hundred and forty tokens: one client reads 4055 positions a second and draws 75.3; two clients read 4260 between them and draw 115.6.

What made it possible is that a column now says which conversation it belongs to as well as where it sits, so a block's cache on the card is one ring per conversation rather than one ring. cmd/golem-server/README.md has the rest.

Qwen3.8 on the card is the exception, and -parallel above 1 is refused there rather than fallen back from. Forty-eight of its sixty-four blocks are delta nets, and a delta net keeps a state matrix a head that every token rewrites rather than a ring indexed by position: there is no ring to cut into slots, and answering two conversations off one state is a wrong answer and not a slow one. On the processor it holds slots like the rest.

⚡ Vulkan GPU

-vulkan puts the whole model on the card. Measured on an RX 9070 XT, the same evening, against llama.cpp's own Vulkan build on the same files:

Reading a prompt, golem is ahead on all five models at 64 and 256 positions, and on the 26B A4B, the 0.6B and Qwen3.8 at 512 as well. On the 26B A4B that is a factor of two and a half at sixty-four positions; on Qwen3.8 the two engines land within a tenth of a per cent of each other at 512, which the section below takes apart op by op.

Generating, golem is ahead on both Gemma models and within two per cent on both Qwen3 ones. Qwen3.8 generates slightly behind due to the heavy per-token state updates of its gated delta net, which cannot be amortised across a batch.

The card holds the whole model: 12.8 GiB for the 26B A4B, which is why sixteen is the smallest card that can run it, and about nine seconds of upload. -vulkan is all or nothing and says so: a card without VK_KHR_shader_integer_dot_product, a machine with no Vulkan loader, a model too large for the card — each is an error at startup rather than a silent half-move. Without the flag, everything runs on the CPU as before.

(See ARCHITECTURE.md for the kernel work behind these numbers.)

🔮 Qwen3.8 drafts its own next token

The Qwen3.8 checkpoint ships a sixty-fifth block: a multi-token-prediction head that guesses the token after the one just decided, from the state the trunk has already computed. Guess right and the next pass verifies two tokens for the price of one. Guess wrong and it costs the pass it rode on and nothing else — every token returned is drawn from the model's own distribution, so this needs none of the accept-reject correction that drafting with a separate model does. The answer is the same with drafting on and off, and a test asserts it.

Qwen3.8 27B, a seventeen-token prompt, greedy:

Seventeen positions is a narrow pass, which is why the card reads this prompt at a hundred and twenty a second and a five-hundred-and-twelve-position one at 1236 — the section below is that curve.

Drafting is refused on the processor, and the reason is one measurement. A speculative step costs a draft plus a pass of two columns, against the one-column pass it hopes to replace:

The card reads a block's weights once whether the pass carries one column or two, so verifying two tokens costs what drawing one did. The processor at this size is bound by arithmetic rather than by reading the weights, so two columns cost two columns — and no acceptance rate can pay for that. qwen35/cost_test.go is where both tables come from.

📈 Reading a prompt on the card

The same bargain reads a prompt. A pass carries up to five hundred and twelve positions, and what one costs says how far that goes:

Thirty-two positions cost less than sixteen, and that is not a misprint: sixteen is the widest the mat-vec builds, and thirty-two is where the tiled products take over.

A prompt reads at 1236 positions a second, against 62 when a pass carried two. llama.cpp's Vulkan build reads the same file on the same card at 1234.67 ± 2.00 (llama-bench -p 512 -r 3). Both benchmarks do the same work — 512 tokens from an empty cache, both warm, both synchronised, load time excluded — with one difference, and it is ours: golem reads every position's hidden state back over the bus where llama.cpp keeps only the last.

Where a pass goes

Both engines report their own GPU timers op by op — vk.Timeline here, GGML_VK_PERF_LOGGER=1 there. One pass of 512 positions of the 27B, in milliseconds:

The card is busy 97.6% of that wall clock — the timer's ticks account for all but 2.4% of it — so nothing measurable is lost between dispatches, and every difference above is a kernel.

Attention is the one line where the gap is a shape rather than a margin: golem dispatches the scores, the softmax and the mix as three passes over memory, where llama.cpp fuses them and keeps the block of scores where it computed it. Sixteen of the sixty-four blocks attend at all, which is why thirty milliseconds buys the other engine so much here and why this is the next line to take.

🧠 Supported Models

In absolute terms, on an i7-9700K with eight threads and Q4_0 weights: Gemma E2B draws 22.6 tokens a second and reads 204; the 12B, 5.0 and 42; the 26B A4B, 13.1 and 51; Qwen3 4B, 14.6 and 110. Pocket TTS speaks at ×2.94 real time in French, ×6.81 in English.

The 0.6B is the one this engine loses, and qwen/README.md says why: at 320 MB the weights fit close enough that the memory bus stops being the limit, and what is left is arithmetic, where llama.cpp's kernels win. This engine is built for the regime where reading the weights is the cost, and it says so where it is not.

👁️ Multimodal (Vision & Audio)

Provide the projector weights, and Gemma can see and hear:

Analyze images:

./golem-cli -model gemma-4-E2B-it-QAT-Q4_0.gguf \
    -mmproj mmproj-gemma-4-E2B-it-QAT-BF16.gguf \
    -image photo.png -p "What is in this picture?"

Transcribe and answer from audio:

./golem-cli -model gemma-4-E2B-it-QAT-Q4_0.gguf \
    -mmproj mmproj-gemma-4-E2B-it-QAT-BF16.gguf \
    -audio question.wav -p "Answer what is asked."

WAV, MP3 and FLAC, at any rate and any number of channels; the front end downmixes and resamples before the encoder sees anything. The 26B's projector carries no audio weights, so that checkpoint sees and does not hear.

The server reads the OpenAI content parts a client sends — image_url for a picture, input_audio for a recording — as a data: URI, as base64, or as a path on the machine it runs on. It does not fetch either over the network: a server that fetches what a prompt names is a server that can be aimed.

🗣️ Speech (Pocket TTS)

go build ./cmd/pocket-tts

./pocket-tts -voice voice.safetensors -o hello.wav "Bonjour le monde."

Clone a voice from a recording, without training and without leaving Go — twenty to thirty seconds of mono 24 kHz is enough:

./pocket-tts -clone someone.wav -save-voice someone.safetensors
./pocket-tts -voice someone.safetensors -o answer.wav "And now I speak in that voice."

🔬 The Method

No layer is deemed correct until its intermediate activations match the reference implementation.

Scripts load the real weights, inject a deterministic input, and write every intermediate quantity into testdata/. The Go tests read those files back, so they need neither Python nor llama.cpp at test time. For gemma/ the reference is llama.cpp itself, instrumented, because a bf16 reference would bury a mistake under its own quantization error; for pockettts/ it is PyTorch, layer by layer, to a few parts in a million end to end.

Every number in this README is a benchmark in this repository, run on the machine named beside it. Nothing is estimated.

🛠️ Project Structure

• cmd/golem-cli, cmd/golem-server, cmd/pocket-tts — the three commands.
• engine/ — reads the architecture out of a GGUF and opens the engine that implements it.
• gemma/, qwen/, qwen35/, pockettts/ — standalone engine implementations; they do not import one another. qwen35/ is Qwen3.8: a package is named for the architecture the GGUF declares, and this checkpoint declares general.architecture = qwen35, as llama.cpp's own models/qwen35.cpp does.
• nn/ & vk/ — the shared kernels: quantized AVX2 and NEON, and Vulkan compute.
• tensors/, token/, chat/, sample/, audio/, imageio/ — the rest of the shared layer.
• ref/ — what recorded each test fixture, and how to record it again.

🤝 Contributing

We want to make Golem the best pure-Go inference engine available. We especially need:

1. ARM benchmarks: the arm64 kernels are correct and tuned by nobody — written and verified under emulation, never once timed on real hardware. Run ./benchmark-arm.sh on Apple Silicon or Graviton and share the results.
2. Bug reports: if a test comparing against PyTorch or llama.cpp fails on your setup, please open an issue.
3. Kernel optimization: help tune the NEON kernels for ARM64.

go build ./...
go test ./...

Weights are not in this repository, and every test that needs one skips cleanly when it cannot find it.

📜 License & Credits

Golem is MIT Licensed.

Standing on the shoulders of giants: llama.cpp & ggml, Kyutai Pocket TTS, Google Gemma.