golem

Gemma, Qwen and Pocket TTS in a single static Go binary.
No Python. No cgo. No GPU required — but with -vulkan it outruns llama.cpp's own Vulkan build, reading prompts and generating both.

A golem is inert matter given a voice. That is what these engines do to a file of weights.

Golem is a set of inference engines written in pure Go. Run Gemma 4, Qwen3, Qwen3.5 and Kyutai Pocket TTS locally with go build, a GGUF file, and your CPU — or your Vulkan GPU.

✨ Features

• Zero Friction: Compiles to a single static binary. No Python environment, no cgo, no runtime to install.
• Pure Go, four dependencies: purego for the Vulkan loader, and three file formats the standard library does not read — WebP, MP3, FLAC. CGO_ENABLED=0 go build ./... passes.
• OpenAI Compatible: Drop-in replacement for OpenAI API clients, tool calls included.
• Multimodal: Text, Vision (images) and Audio (WAV/MP3/FLAC) via Gemma 4.
• Verified, not asserted: no layer is deemed correct until its intermediate activations match llama.cpp or PyTorch, waypoint by waypoint.
• Fast on CPU: keeps pace with llama.cpp on tuned AVX2 kernels — ahead reading prompts, level generating.
• Vulkan GPU: faster than llama.cpp's Vulkan build on Gemma — reading prompts and generating — bound through purego rather than cgo.
• Serves several clients at once: -parallel N holds N conversations and carries a token for each of them through one read of the weights, on the card as well as on the processor.

🚀 Quickstart

Download a GGUF and run. No configuration file — the engine reads general.architecture out of the file and opens whichever engine implements it, so no command here names a model family.

CLI (chat in the terminal)

go build ./cmd/golem-cli

./golem-cli -model gemma-4-E2B-it-QAT-Q4_0.gguf -p "Explain a mutex in one sentence." -stats

Server (OpenAI-compatible API)

go build ./cmd/golem-server

./golem-server -model Qwen3-4B-Q4_0.gguf -addr 127.0.0.1:8080

-parallel N cuts the context into N slots, each holding its own conversation.

👥 Several conversations, one pass

Whatever is waiting at the moment a pass is built goes into that pass. Four clients each wanting a token are four tokens in one read of a gigabyte of weights, rather than four reads — llama.cpp's continuous batching, and it now works on the card as well as on the processor. The context is cut between the slots rather than multiplied, so -context 4096 -parallel 4 is four conversations of 1024 and the memory is what it was.

Gemma 4 26B A4B on an RX 9070 XT, drawing a token for each conversation from 448 positions of context:

Through the API, with prompts of about five hundred and forty tokens: one client reads 4055 positions a second and draws 75.3; two clients read 4260 between them and draw 115.6.

What made it possible is that a column now says which conversation it belongs to as well as where it sits, so a block's cache on the card is one ring per conversation rather than one ring. cmd/golem-server/README.md has the rest.

⚡ Vulkan GPU

-vulkan puts the whole model on the card. Measured on an RX 9070 XT, the same evening, against llama.cpp's own Vulkan build on the same files:

Reading a prompt, golem is ahead on all four models at 64 and 256 positions, and on the 26B A4B and the 0.6B at 512 as well. On the 26B A4B that is a factor of two and a half at sixty-four positions.

Generating, golem is ahead on both Gemma models and within two per cent on both Qwen3 ones. That is new, and what closed it was not a kernel: Gemma caps its logits at thirty, and taking a hyperbolic tangent of a quarter of a million of them on one core cost a third of every token, after the card had already finished. It happens on the card now, at the end of the product that made the number. The Qwen3 models have no softcap, which is why they had no gap to close.

The card holds the whole model: 12.8 GiB for the 26B A4B, which is why sixteen is the smallest card that can run it, and about nine seconds of upload. -vulkan is all or nothing and says so: a card without VK_KHR_shader_integer_dot_product, a machine with no Vulkan loader, a model too large for the card — each is an error at startup rather than a silent half-move. Without the flag, everything runs on the CPU as before.

(See ARCHITECTURE.md for the kernel work behind these numbers.)

🔮 Qwen3.5 drafts its own next token

The Qwen3.5 checkpoint ships a sixty-fifth block: a multi-token-prediction head that guesses the token after the one just decided, from the state the trunk has already computed. Guess right and the next pass verifies two tokens for the price of one. Guess wrong and it costs the pass it rode on and nothing else — every token returned is drawn from the model's own distribution, so this needs none of the accept-reject correction that drafting with a separate model does. The answer is the same with drafting on and off, and a test asserts it.

Qwen3.5 27B A3B, a seventeen-token prompt, greedy:

Drafting is refused on the processor, and the reason is one measurement. A speculative step costs a draft plus a pass of two columns, against the one-column pass it hopes to replace:

The card reads a block's weights once whether the pass carries one column or two, so verifying two tokens costs what drawing one did. The processor at this size is bound by arithmetic rather than by reading the weights, so two columns cost two columns — and no acceptance rate can pay for that. qwen35/cost_test.go is where both tables come from.

🧠 Supported Models

In absolute terms, on an i7-9700K with eight threads and Q4_0 weights: Gemma E2B draws 22.6 tokens a second and reads 204; the 12B, 5.0 and 42; the 26B A4B, 13.1 and 51; Qwen3 4B, 14.6 and 110. Pocket TTS speaks at ×2.94 real time in French, ×6.81 in English.

The 0.6B is the one this engine loses, and qwen/README.md says why: at 320 MB the weights fit close enough that the memory bus stops being the limit, and what is left is arithmetic, where llama.cpp's kernels win. This engine is built for the regime where reading the weights is the cost, and it says so where it is not.

Qwen3.5 reads prompts slowly — 47.6 positions a second on a card against llama.cpp's 615. Generation is ahead; the prompt is not, because a recurrent block carries state from one position to the next and this engine's pass is two positions wide where it wants five hundred. That is the work left on it, and it is named here rather than left out of the table.

👁️ Multimodal (Vision & Audio)

Provide the projector weights, and Gemma can see and hear:

Analyze images:

./golem-cli -model gemma-4-E2B-it-QAT-Q4_0.gguf \
    -mmproj mmproj-gemma-4-E2B-it-QAT-BF16.gguf \
    -image photo.png -p "What is in this picture?"

Transcribe and answer from audio:

./golem-cli -model gemma-4-E2B-it-QAT-Q4_0.gguf \
    -mmproj mmproj-gemma-4-E2B-it-QAT-BF16.gguf \
    -audio question.wav -p "Answer what is asked."

WAV, MP3 and FLAC, at any rate and any number of channels; the front end downmixes and resamples before the encoder sees anything. The 26B's projector carries no audio weights, so that checkpoint sees and does not hear.

The server reads the OpenAI content parts a client sends — image_url for a picture, input_audio for a recording — as a data: URI, as base64, or as a path on the machine it runs on. It does not fetch either over the network: a server that fetches what a prompt names is a server that can be aimed.

🗣️ Speech (Pocket TTS)

go build ./cmd/pocket-tts

./pocket-tts -voice voice.safetensors -o hello.wav "Bonjour le monde."

Clone a voice from a recording, without training and without leaving Go — twenty to thirty seconds of mono 24 kHz is enough:

./pocket-tts -clone someone.wav -save-voice someone.safetensors
./pocket-tts -voice someone.safetensors -o answer.wav "And now I speak in that voice."

🔬 The Method

No layer is deemed correct until its intermediate activations match the reference implementation.

Scripts load the real weights, inject a deterministic input, and write every intermediate quantity into testdata/. The Go tests read those files back, so they need neither Python nor llama.cpp at test time. For gemma/ the reference is llama.cpp itself, instrumented, because a bf16 reference would bury a mistake under its own quantization error; for pockettts/ it is PyTorch, layer by layer, to a few parts in a million end to end.

Every number in this README is a benchmark in this repository, run on the machine named beside it. Nothing is estimated.

🛠️ Project Structure

• cmd/golem-cli, cmd/golem-server, cmd/pocket-tts — the three commands.
• engine/ — reads the architecture out of a GGUF and opens the engine that implements it.
• gemma/, qwen/, qwen35/, pockettts/ — standalone engine implementations; they do not import one another.
• nn/ & vk/ — the shared kernels: quantized AVX2 and NEON, and Vulkan compute.
• tensors/, token/, chat/, sample/, audio/, imageio/ — the rest of the shared layer.
• ref/ — what recorded each test fixture, and how to record it again.

🤝 Contributing

We want to make Golem the best pure-Go inference engine available. We especially need:

1. ARM benchmarks: the arm64 kernels are correct and tuned by nobody — written and verified under emulation, never once timed on real hardware. Run ./benchmark-arm.sh on Apple Silicon or Graviton and share the results.
2. Bug reports: if a test comparing against PyTorch or llama.cpp fails on your setup, please open an issue.
3. Kernel optimization: help tune the NEON kernels for ARM64.

go build ./...
go test ./...

Weights are not in this repository, and every test that needs one skips cleanly when it cannot find it.

📜 License & Credits

Golem is MIT Licensed.

Standing on the shoulders of giants: llama.cpp & ggml, Kyutai Pocket TTS, Google Gemma.