# Development > **Related:** [Access Control](https://wavehouse.dev/access-control.md) · [API Reference](https://wavehouse.dev/api.md) · [Architecture](https://wavehouse.dev/architecture.md) · [Claude Code & AI agents](https://wavehouse.dev/claude-code.md) · [Configuration](https://wavehouse.dev/configuration.md) · [Deployment](https://wavehouse.dev/deployment.md) · [Durability & Storage](https://wavehouse.dev/durability.md) · [Getting Started](https://wavehouse.dev/getting-started.md) · [Ingest Pipeline](https://wavehouse.dev/ingest-pipeline.md) · [Named Pipes](https://wavehouse.dev/pipes.md) · [Behind a reverse proxy](https://wavehouse.dev/reverse-proxy.md) · [TypeScript SDK](https://wavehouse.dev/sdk.md) · [Settings Directory](https://wavehouse.dev/settings-directory.md) · [Upgrading](https://wavehouse.dev/upgrading.md) · [Why WaveHouse?](https://wavehouse.dev/why-wavehouse.md) > **Also:** [HTML version](https://wavehouse.dev/development) · [Docs index](https://wavehouse.dev/llms.txt) --- Everything you need to build, test, lint, and contribute to WaveHouse — from first-clone to hot-reload dev server to full end-to-end SDK tests. If you're only trying the product, start with the [Getting Started](/getting-started) guide instead. ## Prerequisites You need these on your `PATH` before any `make` recipe will work end-to-end: | Tool | Required version | Why | Install | | ---- | ---------------- | --- | ------- | | **Go** | Any Go ≥ 1.21; `make` runs the version `go.mod` pins, its `toolchain` line if it has one, else its `go` line (currently 1.26.9), downloading it once if yours differs (needs network). `make GOTOOLCHAIN=local …` uses yours instead (an environment value is ignored): it needs a Go ≥ that version, and `make lint-go` needs `go.mod`'s minor | Compiles `cmd/wavehouse`; also runs the pinned `tool` deps (`gotestsum`, `gofumpt`, `goimports`, `govulncheck`, `go-test-coverage`, `gocover-cobertura`, `deadcode`, `gsa`, `goda`) via `go tool` | [go.dev/dl](https://go.dev/dl/) | | **GNU Make** | **4.0+** | The Makefile uses `--output-sync=target` (Make 4 only) and bash-pinned recipes. macOS ships with GNU Make 3.81, which is **not supported**: it silently ignores `--output-sync`, so the parallel output of `make verify` / `make ci` interleaves line by line and a failure can scroll away behind other jobs' output | macOS: `brew install make` then use `gmake` or put `$(brew --prefix make)/libexec/gnubin` on your PATH. Linux: usually already installed | | **bash** | 4+ recommended | Recipes are pinned to `bash`; the helper scripts under `scripts/` use `set -euo pipefail` and bash arrays | macOS default is bash 3.2 (works for current recipes, but `brew install bash` is safer); Linux distros ship 4+ | | **Docker** *(or Podman)* | Engine 20.10+ with the Compose **v2** plugin (`docker compose`, no hyphen) | Compose stacks under `deployments/compose/`; the E2E and integration suites boot ClickHouse and a Redis via testcontainers (no compose file), the integration suite also dynamodb-local, and the integration suite also runs the shared cache backend against Redis, Valkey, Dragonfly (pulled from `docker.dragonflydb.io`) and a one-node Redis Cluster | [Docker Desktop](https://docs.docker.com/get-docker/), [colima](https://github.com/abiosoft/colima), or [Podman](https://podman.io) with `podman-compose` / the `podman compose` plugin. The testcontainers Go library also honors `DOCKER_HOST` for rootless Podman setups | | **Node.js** | 22 LTS — pinned via `.nvmrc` at the repo root | Runtime for pnpm and the Vitest suites. Pinned to match CI (`setup-node` uses 22) and to avoid Node-major surprises; older Vitest versions in this repo were known to crash on Node 26 with a V8 heap-allocation abort | [nodejs.org](https://nodejs.org/) or `nvm use` / `fnm use` / `volta` (all read `.nvmrc`) | | **pnpm** | 11.21+ (pinned via `packageManager` in the root `package.json`) | Package manager for the TypeScript SDK, E2E test harness, and docs site (managed as a single pnpm workspace from the repo root); `make tools`, `make verify`, `make lint`, `make fmt`, `make fix`, `make ci`, `make build-ts`, `make dev-ts`, `make test-ts`, `make test-e2e`, `make test-all`, `make check-docs`, `make build-docs`, `make build-all`, `make dev-docs`, `make preview-docs` all shell out to `pnpm` (`make tools` itself runs `pnpm install`, and `make verify` is what the pre-commit hook runs, so a Go-only contributor needs pnpm too) | `corepack enable && corepack prepare pnpm@11.21.0 --activate` (recommended), or `npm i -g pnpm` | | **git** + **curl** + **jq** | any recent | `git` for source + version metadata in builds; `curl` is used by the Makefile to fetch the pinned `golangci-lint` and `shellcheck` binaries into `.bin/` and, for `make build-ts`, the oldest Node that `engines.node` admits; `jq` is needed by `make verify` (the hook tests) and by the Claude Code hooks | usually preinstalled (`jq` ships with macOS 15+); otherwise `apt install jq` / `brew install jq` | ### Auto-installed by `make tools` Run `make tools` once after cloning to populate everything that doesn't have to be on your PATH: - **`golangci-lint` v2.11.4** → installed to `.bin/_/` (version-pinned in the Makefile; bumping the version triggers a reinstall). Not in `go.mod` because its dependency tree conflicts with the main module. - **`misspell` v0.8.0, `shellcheck` v0.11.0, `actionlint` v1.7.12** → installed to `.bin/_/`; they back `make lint-prose`, `make lint-sh`, and `make lint-gha`. `make tools` also points `core.hooksPath` at `.githooks/`, which is what installs the pre-commit and pre-push gates (`git config --unset core.hooksPath` turns them off for every worktree of the clone, and the next `make tools` turns them back on; use `--no-verify` for a one-off). - **`air` v1.65.1** → installed to `.bin/_/` via `go install`; used by `make dev` for hot-reload. Same exclusion principle as `golangci-lint` — air's transitive deps (Hugo, Sass libs) would bloat `go.sum`. - **Go `tool` deps** (`gotestsum`, `gofumpt`, `goimports`, `govulncheck`, `go-test-coverage`, `gocover-cobertura`, `deadcode`, `gsa`, `goda`) — pinned in `go.mod` via native `tool` directives (Go 1.24+), invoked with `go tool `. `make tools` runs `go mod download` so they're cached; they compile lazily on first invocation. - **pnpm deps** for the repo root plus `clients/ts/`, `tests/e2e/sdk/`, and `docs/` (via `pnpm install --frozen-lockfile`). The root workspace is where the repo-wide linters live — **Biome** (JS/TS/JSON) and **markdownlint-cli2** (Markdown/MDX, including the repo-local `WH001`/`WH002` rules). Neither is ever run from a global install: `make lint` / `make fix` shell out to `pnpm -w run`, and every target that needs them declares the `pnpm-install` prerequisite, so a clean clone is linting correctly after `make tools` with nothing else on your PATH. `make tools` runs only the pnpm install; `make build-ts` also downloads the oldest Node that `engines.node` admits (a 45–51 MB tarball, about 175 MB unpacked, once, into `.bin/`, verified against a pinned sha256) to smoke-load the built SDK entry points with `make smoke-ts-dist`; the Playwright Chromium binary (~130 MB) is fetched on-demand by `make build-docs` / `make dev-docs` via the internal `install-playwright-docs` target, so Go-only contributors don't pay the download cost. When you do hit `build-docs` / `dev-docs`, Chromium is required by two parts of the docs *build*: `rehype-mermaid` (SVG diagram rendering) and the `diagramPng` integration from `@wave-rf/astro-themed-mermaid/png` (wired in `docs/astro.config.mjs`), which rasterizes each diagram to light/dark PNGs (a solid surface-card variant plus a transparent-background variant for slide decks) at `astro:build:done` for the Copy/Download buttons in the `diagramZoom` lightbox. Both reuse the same Playwright Chromium, as does the manual `docs/scripts/screenshot.mjs` QA helper. `starlight-links-validator` runs under `build-docs` / CI only — the `dev-docs` watch loop skips it so a mid-edit dangling link doesn't fail every rebuild (CI still enforces link validity before merge; run `MD_ROUTER_DEV_STRICT=1 make dev-docs` to keep the validator on locally). The `--with-deps` flag (which apt-installs Chromium's system libraries: `libnspr4`, `libnss3`, etc.) is only added when `$CI` is set, so contributor laptops don't get an unexpected `sudo` prompt. On Linux dev machines without those libs already present, run `pnpm --filter wavehouse-docs exec playwright install-deps chromium` once manually. The docs site is a pnpm workspace package (`wavehouse-docs`); the root Makefile drives it directly via `pnpm --filter` (no sub-Makefile) — the `*-docs` targets show up in `make help`. It is also a real `@wavehouse/sdk` consumer, but of the **published** package: the landing page's live demo takes the SDK from the registry, pinned in `docs/package.json`, not from `clients/ts/`. That is deliberate — the demo streams against a separately-deployed backend, so binding it to the workspace would ship an unreleased wire format to the live site the moment a change merged ([#568](https://github.com/Wave-RF/WaveHouse/issues/568)). The pin is raised on purpose when a new SDK release is tagged. Nothing in the docs build needs `build-ts`, including when you drive Astro directly through pnpm (e.g. `pnpm --filter wavehouse-docs run start`). pnpm's hoisted fallback is off; see [Updating Dependencies](#updating-dependencies). ### Verify your setup ```bash go version # go1.21+ (make runs go.mod's version itself) make --version # GNU Make 4.x docker compose version node --version # v22.x (matches .nvmrc and CI) pnpm --version # 11.21+ jq --version ``` If any of those are wrong/missing, the Makefile recipes will fail or misbehave confusingly (e.g. a failure buried in interleaved parallel output on Make 3.81; `pnpm: command not found` on `make tools`). ### Optional but recommended | Tool | Why | Install | | ---- | --- | ------- | | **[Claude Code](https://claude.com/claude-code)** | The repo ships team-wide configuration in `.claude/` — slash commands, subagents, hooks, status line. See [Claude Code & AI agents](/claude-code) for setup. | `brew install --cask claude-code` (macOS) or follow [official install](https://code.claude.com/docs/en/quickstart) | | **[worktrunk](https://worktrunk.dev)** | Wraps `git worktree` for parallel-agent workflows. Project hooks live in `.config/wt.toml` (auto-runs `make tools` on new worktrees, `make verify` on pre-merge). | `brew install worktrunk && wt config shell install` | ## Quick Start This is the fastest way to get a fully functional local environment: ```bash # 1. Clone and bootstrap (Go modules + golangci-lint + pnpm deps) git clone https://github.com/Wave-RF/WaveHouse.git cd WaveHouse make tools # 2. Start ClickHouse (the only external dependency) docker compose -f deployments/compose/dependencies.yaml up -d clickhouse # 3. Create a table in ClickHouse docker compose -f deployments/compose/dependencies.yaml exec clickhouse \ clickhouse-client --query " CREATE TABLE IF NOT EXISTS clicks ( page String, button String, score Float64, received_timestamp DateTime64(3, 'UTC') DEFAULT now64(3, 'UTC') ) ENGINE = MergeTree() ORDER BY (page) " # 4. Run with hot-reload (recompiles on every .go file save) make dev ``` WaveHouse is now running at `http://localhost:8080` in standalone mode with: - **Embedded NATS** (JetStream) — no external MQ needed - **In-process cache** (Ristretto, `cache.backend: local`) — no external cache needed; to try the shared one, start Redis with `docker compose -f deployments/compose/dependencies.yaml --profile redis up -d` and set `WH_CACHE_BACKEND=redis WH_CACHE_REDIS_ADDRS=localhost:6379` - **Trial policy** — the dev settings directory `./settings` is seeded on first run with the compose stack's permissive `public` policy, so tokenless requests to the demo tables work (see [Test the API](#test-the-api)) - **Dedup disabled** by default — no Pebble needed - **Schema discovery** — automatically finds your ClickHouse tables ### Test the API On first run `make dev` seeds the gitignored `./settings` directory with `wavehouse bootstrap` and copies in the compose stack's trial policy (`deployments/compose/settings/policies.json` + `roles.json` — the `public` role: read/write `clicks`/`events`, no token). WaveHouse is otherwise fail-closed (the bootstrap seed ships no policy), so a `./settings` you emptied by hand denies every request until you put a policy back. Edits to any file in `./settings` hot-reload without a restart. The tokenless data-plane calls work (create a `clicks` table first — see the [Getting Started](/getting-started) walkthrough): ```bash # Ingest an event curl -s -X POST "http://localhost:8080/v1/ingest?table=clicks" \ -H "Content-Type: application/json" \ -d '{"page": "/home", "button": "signup", "score": 42.5}' # → {"ok":true} # Query it back (wait ~5s for the batch flush) curl -s -X POST "http://localhost:8080/v1/query?table=clicks" \ -H "Content-Type: application/json" \ -d '{"columns": ["page", "button", "score"], "limit": 10}' # Open an SSE stream for a specific table (Ctrl+C to stop) curl -N "http://localhost:8080/v1/stream?table=clicks" # With gap-fill (replays events since the given timestamp, then switches to live) curl -N "http://localhost:8080/v1/stream?table=clicks&since=2026-03-24T11:00:00Z" # Liveness / readiness (no auth required) curl http://localhost:8080/livez # → {"status":"ok"} curl http://localhost:8080/readyz # → {"status":"ready"} ``` The admin surface — `/v1/ops/schema`, `/v1/ops/query` (raw SQL), `/v1/ops/dlq/stats` — needs the **admin** role, which the `public` trial role doesn't have. Mint an admin JWT (see [Validating tokens](#validating-tokens) below) and pass it: ```bash curl -s http://localhost:8080/v1/ops/schema -H "Authorization: Bearer $TOKEN" | jq curl -s -X POST http://localhost:8080/v1/ops/query -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" -d '{"sql": "SELECT * FROM clicks LIMIT 10"}' curl -s http://localhost:8080/v1/ops/dlq/stats -H "Authorization: Bearer $TOKEN" ``` ### How `make dev` works `make dev` is a one-stop convenience target for backend and frontend development. The recipe is essentially: ```make dev: deps-up $(AIR) air -c .air.toml ``` `deps-up` runs `docker compose ... up -d --wait clickhouse`, which blocks until the ClickHouse container's `/ping` healthcheck flips to healthy. `$(AIR)` lazily installs air to `.bin/_/` if missing. Then air takes over: it watches `cmd/`, `internal/` and `pkg/` (the `.go` and `.yaml` files within them), rebuilds `tmp/wavehouse` on change, and restarts the binary. Boot config is **not** hot-reloaded (the settings directory it points at is — see [Enable Dedup](#enable-dedup-optional)): `make dev` runs the binary with `WH_CONFIG=.config.local.yaml` — a gitignored personal copy seeded **once** from `config.yaml` on first run (it won't re-copy if it already exists). So to change dev config, edit `.config.local.yaml` (not `config.yaml`) and restart `make dev`; air watches neither root file. The flip side: when `config.yaml` gains a key your copy lacks, your copy is stale — `make dev` warns when `config.yaml` is newer than `.config.local.yaml`, and the fix is to port the change over or delete `.config.local.yaml` to re-seed it. `settings.dir` is the one key that will stop the server outright if it's missing (it's required with no default), so a checkout whose local copy predates it fails at config validation until re-seeded. `air` is pinned to a specific version and installed via `go install` rather than a `go.mod` tool directive — its transitive deps (Hugo, godartsass, Sass libs) would bloat `go.sum` for everyone. Same exclusion principle as `golangci-lint`. **While `make dev` is running you get:** - WaveHouse on `http://localhost:8080` with the default allow-all CORS posture, so a browser-based app on any localhost port can hit the API directly. - The trial `public` policy ships in `./settings` (see [Test the API](#test-the-api)); its `auth.jwt_secret` is empty, so no token validates until you set one there (see [Validating tokens](#validating-tokens)). - ClickHouse on `http://localhost:8123` (HTTP) and `localhost:9000` (native protocol), Compose project name `wavehouse-dev` so containers/volumes are namespaced. - Go's pprof endpoints on `http://localhost:6060/debug/pprof/` (`go tool pprof http://localhost:6060/debug/pprof/heap`): `make dev` sets `WH_PROFILING_ENABLED=true` unless it is already set in your environment, so `WH_PROFILING_ENABLED=false make dev` turns them off. The listener's address is `profiling.addr` in `.config.local.yaml` (see [Configuration → Profiling](/configuration#profiling)). - Hot reload: editing any `.go` file under `cmd/`, `internal/` or `pkg/` triggers a debounced rebuild + restart. Boot config isn't hot-reloaded (settings-directory files are) — `make dev` loads `.config.local.yaml` (a gitignored copy seeded once from `config.yaml`), so edit `.config.local.yaml` and restart to apply config changes. Air's stdout/stderr stream live so you see compile errors and server logs in the same terminal. ### Dev convenience targets These are the small targets behind `make dev` — useful directly when you want to run WaveHouse outside of air (e.g. `make settings/config.json && make build && ./bin/wavehouse`), or when you need to poke at ClickHouse: | Target | What it does | | ------ | ------------ | | `make deps-up` | Start ClickHouse and block until healthy. Idempotent. | | `make deps-down` | Stop ClickHouse. Data volume is preserved. | | `make deps-logs` | `docker compose logs -f clickhouse` (Ctrl+C detaches; container keeps running). | | `make deps-shell` | Drop into a `clickhouse-client` REPL on the running container. | | `make deps-wipe` | Stop ClickHouse **and destroy its data volume**. Use when you want a clean schema. | | `make clean-all` | Nuclear option — every `make` artifact + dev/E2E containers + volumes + `data/`. | **Stopping `make dev`**: `Ctrl+C` stops air, which propagates SIGINT to WaveHouse for a graceful shutdown (NATS JetStream flush, etc.). ClickHouse stays up — re-running `make dev` is fast because the volume is preserved. Use `make deps-down` or `make deps-wipe` to stop ClickHouse explicitly. ### Running with observability With `otel.enabled` on, WaveHouse exports standard OpenTelemetry (OTLP) data to the endpoint in `OTEL_EXPORTER_OTLP_ENDPOINT`. Rather than coupling a heavy observability database stack to the dev server, we provide three lightweight, single-container dashboard options. You run these in a separate terminal tab alongside `make dev` or your test suites (`make test-e2e`). They block the terminal and stream logs; simply press `Ctrl+C` to instantly tear them down and clean up the container. | Target | What it does | UI URL | | ------ | ------------ | ------ | | `make obs-aspire` | Boots the Aspire dashboard. Extremely fast, in-memory only, and requires no login. Ideal for quick trace and log debugging. | `http://localhost:18888` | | `make obs-grafana` | Boots Grafana LGTM (Loki, Grafana, Tempo, Prometheus). Pre-configured to bypass login. Best for advanced UI charting and trace-to-log correlation. | `http://localhost:3000` | | `make obs-front` | Boots OTel-Front for a basic, alternative trace viewer. | `http://localhost:8000` | **Typical Workflow:** 1. Open Tab 1: run `make obs-aspire` (UI opens automatically) 2. Open Tab 2: run `make test-e2e`, which enables OTel and points it at `http://127.0.0.1:4317` for you, or `WH_OTEL_ENABLED=true OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 make dev` (`make dev` leaves OTel off, and the SDK's unset default is TLS, which these local receivers reject) 3. View traces, metrics, and logs flowing into the UI instantly. No accounts or auth tokens required. ### Using the SDK against `make dev` There's no bundled playground — point an `@wavehouse/sdk` client at your local server (`baseURL: "http://localhost:8080"`); the trial policy `make dev` seeds into `./settings` authorizes the demo-table requests: :::caution[For **streaming**, use `@wavehouse/sdk` 0.2.0 or later against a local server] A `make dev` server speaks the v2 SSE wire — an `event: schema` frame plus positional rows — which `@wavehouse/sdk` speaks from 0.2.0; the 0.1.x line speaks the pre-v2 one. Point a 0.1.x client at `/v1/stream` and every event arrives with `data: undefined`, with no `error` callback and nothing to catch, while REST and structured queries keep working — which makes it easy to misread as a server bug. Use 0.2.0 or later, the in-tree SDK (`make build-ts`) or `@wavehouse/sdk@dev` for streaming work. See [Transport Behavior](/sdk/streaming#transport-behavior), which carries the full danger note on upgrading the SDK and the server together. ::: ```bash make dev ``` See the [SDK guide](/sdk) for the client API and examples. Frontend devs running their own dev server (Vite, Next.js, etc.) can `import { createClient } from '@wavehouse/sdk'` and point `baseURL: 'http://localhost:8080'`; CORS is permissive so cross-origin browser requests just work. ### Validating tokens There is no auth on/off switch — the JWT middleware always runs, but authorization is the policy's job (an empty `policies.json` adopts no policy and denies every token-based caller, admins included — only the operator key below still reaches the admin surface). To exercise token auth in dev, set a known secret in `./settings/config.json` (a settings key, so it reloads live under a running `make dev`) — the trial policy's `admin_role` defaults to `admin`, so a JWT with `role: admin` unlocks the admin surface: ```json "auth": { "jwks_url": "", "jwt_secret": "my-secret", "role_claim": "role" } ``` The **operator key** is a non-JWT alternative: set one and send it in an `Authorization: Operator ` header (or the `X-Operator-Key` alias). Even with no policy adopted it reaches the **admin surface** — enough to trigger a settings reload over HTTP after fixing `policies.json` (the break-glass path). Once a policy is adopted, the key's role resolves to `admin`, so it then has full data-plane access too (pipes, queries, streaming, ingest) — handy for trialing without minting a JWT: ```bash WH_AUTH_OPERATOR_KEY=dev-operator-key make dev # ...then, in another shell — the admin surface works even with no policy adopted: curl -X POST -H "Authorization: Operator dev-operator-key" http://localhost:8080/v1/ops/settings/reload # the X-Operator-Key alias works too: curl -H "X-Operator-Key: dev-operator-key" http://localhost:8080/v1/ops/pipes ``` Then mint a token (role == the policy `admin_role`) and call an admin endpoint: ```bash # Using jwt-cli (https://github.com/mike-engel/jwt-cli) export TOKEN=$(jwt encode --secret "my-secret" '{"role": "admin", "exp": 9999999999}') curl -s -X POST http://localhost:8080/v1/ops/query \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"sql": "SELECT * FROM clicks LIMIT 10"}' ``` ### Enable Dedup (Optional) `make dev` seeds `./settings` (gitignored) from the embedded seed on first run and points `settings.dir` at it; the seed ships with `dedupe.enabled: false`. Flip it there — never in the checked-in `pkg/settings/seed`, which is what `wavehouse bootstrap` writes for everyone: ```bash # ./settings already exists after the first `make dev` (or: make settings/config.json) # set "enabled": true under "dedupe" in ./settings/config.json ``` The key hot-reloads, so once the server is running you can toggle it by editing `config.json` — no restart. Records dedupe on their `event_id` field by default; the same file overrides the field globally or per table (see [Settings Directory — Deduplication](/settings-directory#deduplication)). Then include the dedup field in your ingest body: ```bash curl -s -X POST "http://localhost:8080/v1/ingest?table=clicks" \ -H "Content-Type: application/json" \ -d '{"event_id": "550e8400-e29b-41d4-a716-446655440001", "page": "/home"}' # → {"ok":true} # Same event_id again → deduplicated curl -s -X POST "http://localhost:8080/v1/ingest?table=clicks" \ -H "Content-Type: application/json" \ -d '{"event_id": "550e8400-e29b-41d4-a716-446655440001", "page": "/home"}' # → {"duplicate":true} ``` ### Using an .env File ```bash # .env — boot config only (the operator key, sizing, exporters); the # ClickHouse wiring and the token verifier are settings-directory keys export WH_AUTH_OPERATOR_KEY= ``` Then: ```bash source .env make settings/config.json # seeds the gitignored ./settings that config.yaml points at (no-op if it exists) go run ./cmd/wavehouse ``` ## Building ```bash # Build the binary to bin/ make build # Build individual binaries go build -o bin/wavehouse ./cmd/wavehouse ``` ## Running Modes at a Glance | What you want | Command | | ------------- | ------- | | Hot-reload standalone dev server | `make dev` | | Standalone binary (default config) | `make settings/config.json && make build && ./bin/wavehouse` | | Standalone via Docker Compose | `docker compose -f deployments/compose/standalone.yaml up -d` | | Infrastructure deps only (ClickHouse) | `docker compose -f deployments/compose/dependencies.yaml up -d clickhouse` | ## Testing ### How It Works All **Go** test commands use [gotestsum](https://github.com/gotestyourself/gotestsum) for pytest-style colored output with pass/fail icons, durations, and a summary. Tool versions are pinned in `go.mod` via `tool` directives — the Makefile invokes them as `go tool `, so no global installation is needed. All tests run with Go's **race detector** (`-race`) enabled by default. WaveHouse is highly concurrent (NATS consumers, singleflight caching, SSE hubs) — the race detector catches data races that would panic in production. ### Quick Reference ```bash # Prefix any test target with V=1 for verbose output, e.g. `V=1 make test` # Unit tests (compact output) — alias for `test-unit` make test # Run specific test(s) make test ARGS="-run TestValidate" # Go integration tests (requires Docker) make test-integration # SDK vitest unit tests + coverage + gate against suites.ts-unit # (`make cov` auto-merges ts-unit + ts-e2e — no separate command) make test-ts # E2E SDK suite against bin/wavehouse-cov make test-e2e # All four suites sequentially + merged coverage make test-all # Full CI: parallel verify + builds (Go + SDK + docs) + test + test-ts, # then test-integration + test-e2e + cov make ci # Merge available covdata + gate against total threshold make cov ``` Each test target writes `covdata` to `tmp/coverage//data/`, renders a textfmt + HTML report, and gates against the per-suite threshold in `.testcoverage.yml`. `make cov` merges whichever suites have run and gates against the total. **Verbose output**: Use `V=1` to switch from compact `testdox` format to full verbose output. This is a standard Makefile convention (`make test -v` can't work because `-v` is a `make` flag). **Extra flags**: All test targets accept `ARGS="..."` for additional `go test` flags (e.g., `-run`, `-count`, `-timeout`). **Time limits**: `make test-unit` gives each unit package 15s (`UNIT_TIMEOUT`), the budget it has to fit when the suite runs on its own, as it does in the unit job in CI. `make ci` gives it 60s (`CI_UNIT_TIMEOUT`) in its parallel phase, where the suite shares every core with the lint and build jobs, so a package over the budget can pass `make ci` and still fail in CI. `tests/integration` gets 900s (`INTEGRATION_TIMEOUT`). Each is a Makefile variable, so a slower machine can raise one for a run: `make ci CI_UNIT_TIMEOUT=90s`. **Note on timing**: gotestsum's `DONE ... in X.XXXs` reports pure test execution time. The total wall time includes Go compiling all packages — the first run compiles everything (~15s), subsequent runs use the build cache (~1s). **Build cache across worktrees**: `make` exports `GOFLAGS=-trimpath` (your own `GOFLAGS` are kept), so compiled packages don't embed the checkout's path and every worktree of the same source shares one build cache entry per package and variant, instead of building its own. Opt out with `GOFLAGS=-trimpath=false make …`. `make build` is the exception: its debug binary keeps absolute source paths, so a debugger finds the files. Release and coverage binaries and test binaries built through `make` are trimmed; their frames read `github.com/Wave-RF/WaveHouse/internal/…` (standard-library frames are plain import paths such as `runtime/proc.go`), so map the module path to your checkout, for example, at the `(dlv)` prompt, `config substitute-path github.com/Wave-RF/WaveHouse /absolute/path/to/your/checkout` (or a `substitute-path` entry with `from` and `to` in `~/.config/dlv/config.yml`, or your editor's `substitutePath`). Code that needs a file from the repository finds it from the working directory, never from `runtime.Caller`, whose paths are module-relative under `-trimpath`. ### Running `make ci` on another machine `make ci-remote HOST=` runs `make ci` for your `HEAD` commit on another machine and streams its output back, so the heavy suites can run off your laptop, or on another OS or architecture than yours (CI runs on Linux amd64). ```bash make ci-remote HOST=build-box # any ssh destination: user@host, a ~/.ssh/config alias make ci-remote HOST=docker:wavehouse-dev # a running local container or devcontainer make ci-remote HOST=build-box KEEP=1 REMOTE_DIR=/scratch/wavehouse-ci ``` Only the commit travels, so the worktree must be clean: `HEAD` is streamed as a `git bundle` over the ssh or `docker exec` connection, and nothing is pushed. The host checks it out in a fresh directory under `REMOTE_DIR` (default `.cache/wavehouse-ci`; a relative path is under the host's home), runs `make tools && make ci` there, and removes the directory afterwards unless `KEEP=1`. After the local `==> make ci on …` line, the first line the host prints is its `uname -sm`, and each run keeps a copy of its log in `tmp/ci-remote--.log`, so two runs of one tree, say on two architectures at once, don't overwrite each other. If `make ci-remote` is stopped, by Ctrl-C or a signal, the host stops its `make ci` within a second or two: the host's end of the connection stays open only while the local command runs, and the host stops the run when it closes. The host needs what a local `make ci` needs: bash, git, curl, jq, GNU Make 4+, Go, Node and pnpm from [Prerequisites](#prerequisites), network access for `make tools`, and a Docker daemon for the integration and e2e suites. Over ssh the commands run in a non-interactive login shell, so the tools must be on the `PATH` the host's profile sets; in a container they must be on the container's own `PATH`, and the container must reach a Docker daemon (a mounted socket or Docker-in-Docker). If `make ci` runs inside a container on the host's network, testcontainers may try to reach the suites' containers through the Docker bridge gateway and time out; setting `TESTCONTAINERS_HOST_OVERRIDE=localhost` in the host's environment points it at the published ports instead. A Linux host without Chromium's system libraries needs them installed once for the docs build, as Prerequisites describes (`pnpm --filter wavehouse-docs exec playwright install-deps chromium`). **How the pre-push gate treats it.** The gate's meaning doesn't change: a code push still needs `tmp/ci-passed-tree-`, which says `make ci` passed for exactly that tree. `make ci-remote` writes it (and the verify marker) only if the host exits 0, the tree the host reports having checked out equals your `HEAD^{tree}`, `make ci` left the host's checkout clean, and your worktree is still clean and on that tree when the run ends; otherwise it writes nothing. Beside the marker, `tmp/ci-passed-tree-.provenance` records the host, its OS and architecture, the start and end times and the log path, and the pre-push hook prints `make ci passed on (/)` from it. A marker from a local `make ci` has no provenance file and works exactly as before. ### Test Structure | Category | Location | Docker? | Command | | -------- | -------- | ------- | ------- | | Unit tests | `internal/**/*_test.go`, `pkg/**/*_test.go` | No | `make test` | | SDK unit tests | `clients/ts/src/**/*.test.ts` | No | `make test-ts` (always includes coverage + gate) | | Integration tests (Go) | `tests/integration/*_test.go`, `internal/cache/*_integration_test.go`, plus `internal/mq/natsspike` and `internal/mq`'s integration-tagged tests | Yes | `make test-integration` | | E2E tests (SDK) | `tests/e2e/sdk/*.test.ts` | Yes | `make test-e2e` | - **Unit tests** live beside the code they test (e.g., `internal/discovery/discovery_test.go`). They use mocks or embedded NATS (in-process, no Docker needed). - **Integration tests** use the `//go:build integration` build tag. In `tests/integration`, `TestMain` starts one ClickHouse testcontainer and a dynamodb-local one (for the DynamoDB dedupe backend's tests), and boots the production wiring against it through `app.New` (embedded NATS, ingest worker, sweeper, hub, the API server on a random loopback port); tests reach it via `env(t)` and create their own tables. `TestNATSBackend_EndToEnd` also starts a NATS container configured from `deployments/nats/values.yaml`, applies `deployments/nats/jetstream.yaml` to it through `internal/mq/natstest`, and boots two processes on `mq.backend: nats` against it. `TestMain` also builds the `wavehouse` binary while the containers start, so the build is not charged to the `-timeout`: the `TestRoles_*` tests run it as separate OS processes (two API, two then three ingest, one killed with `SIGKILL`) over NATS, Redis and dynamodb-local, and read each ingest process's shard ownership from its `/metrics`. DLQ tests use `assert.Eventually` with a 30-second timeout for the 5-second ingest worker batch window. `internal/cache`'s integration tests start their own containers instead — Redis, Valkey, Dragonfly and a one-node Redis Cluster — for the shared backend. `shared_cache_test.go` starts its own Redis testcontainer per test (`startRedis`) and boots extra, independent `cache.backend: redis` instances over that same ClickHouse (`bootRedisApp`), to exercise the cache shared across processes rather than one package in isolation. `make test-integration` runs the suite in two parts at once, and CI runs each part as a job of its own: `make test-integration-app` runs `tests/integration`, and `make test-integration-backends` runs the backends' own suites. The backends part also runs `internal/mq/natsspike`, which pins the nats-server behavior the external-NATS topology depends on, against an in-process server with no Docker. It lives under `internal/mq` because only that tree may import NATS, and it runs here rather than in the unit suite because each test takes seconds and the unit suite has a 15-second limit per package. For the same reason the external NATS broker's tests (`internal/mq/external*_test.go`, including its run of the `mqtest` conformance suite) and the NATS KV lease tests (`internal/mq/lease_test.go`, including their run of the `coordtest` conformance suite) carry the `integration` tag inside `internal/mq`, and the backends part runs the tests those tagged files declare (`scripts/ci/tagged-tests.sh`), whatever their names, so the package's untagged tests stay in the unit suite alone. In `tests/integration`, most of the slow tests that bring up their own servers call `t.Parallel()` (never one that touches process-wide state, such as `t.Setenv` or `logtest.Capture`), and the app part runs with `-parallel 8`, since those tests mostly wait. - **Images** come from Docker Hub, except Dragonfly's, and Docker Hub limits anonymous pulls per IP address. CI points each container job's Docker daemon at `mirror.gcr.io`, a cache of Docker Hub, before the first pull, with Docker Hub as the fallback (invariant 8 of [`.github/workflows/README.md`](https://github.com/Wave-RF/WaveHouse/blob/main/.github/workflows/README.md)). A local run that fails with `toomanyrequests` can do the same: add `"registry-mirrors": ["https://mirror.gcr.io"]` to the Docker daemon's configuration, merging it into the existing JSON object, and restart the daemon (Docker Desktop: Settings → Docker Engine, then Apply & restart; Linux: `/etc/docker/daemon.json`, then `sudo systemctl restart docker`; rootless Docker: `~/.config/docker/daemon.json`, then `systemctl --user restart docker`), or run `docker login`. Shared test utilities live in `internal/testutil/`. The packages log through `slog.Default()`, so tests reach log output through `internal/testutil/logtest`: `logtest.Silence()` in a package's `TestMain` discards it, and `logtest.Capture(t, level)` routes it to a buffer for a test that asserts on log lines — such a test must not call `t.Parallel()`, because the default logger is process-wide. A test that starts the embedded broker keeps its store in `internal/testutil/storedir`'s `storedir.New(t)` rather than a bare `t.TempDir()` (`testutil.NewEmbeddedMQ` does): the NATS server can finish writing a consumer's state after `Close` returns, which fails `t.TempDir`'s one-shot removal, and `storedir` removes the store again until those writes have landed ([#442](https://github.com/Wave-RF/WaveHouse/issues/442)). ### Adding New Tests - **Unit test for `internal/foo/`** → create `internal/foo/foo_test.go` (same package). - **Integration test needing Docker** → add a subtest under `tests/integration/` (e.g. a new file with `//go:build integration`). A test of one package against its own external server — the shared cache backend against Redis, Valkey and Dragonfly containers — lives beside the package instead (`internal/cache/redis_integration_test.go`, same build tag), and the package joins a part of the suite in the `Makefile` (`INTEGRATION_APP_PKGS` or `INTEGRATION_BACKEND_PKGS`): every part fails while a package with `integration`-tagged tests is in none of the listed parts, while a `test-integration-` target is missing from `INTEGRATION_PARTS`, or while a listed part has no target. A package listed there runs all its tests, untagged ones too (`internal/cache` does); one whose untagged tests belong to the unit suite alone instead gets a run of its own, selected by `scripts/ci/tagged-tests.sh run`, as `internal/mq` has in `test-integration-backends`. - **E2E test via SDK** → add a `tests/e2e/sdk/*.test.ts` file. These tests exercise the full pipeline (ingest → ClickHouse → query) through the TypeScript SDK. Run with `make test-e2e`. - **Test helpers** → add to `internal/testutil/` (Go) or `tests/e2e/sdk/helpers.ts` (E2E). ### E2E Tests via SDK The primary E2E integration test suite lives in `tests/e2e/sdk/`. It uses the TypeScript SDK as the test harness — every ingest→query test simultaneously validates the full Go backend pipeline and confirms SDK compatibility. **Architecture**: - `scripts/orchestrator` — the E2E entrypoint behind `make test-e2e`: it starts a clean ClickHouse **testcontainer** and a Redis one (the fixture's shared cache, `cache.backend: redis`) per run, launches the `wavehouse-cov` binary on a random free port, runs the SDK suite against it, then SIGINTs the binary to flush coverage. No Compose file is involved. CI runs the exact same path. - `tests/e2e/sdk/setup.ts` — `globalSetup`. Probes the `CLICKHOUSE_URL` / `WAVEHOUSE_URL` the orchestrator injects, creates the per-suite tables, refreshes the schema, and writes the baseline policy into the run's settings directory (adopted via `POST /v1/ops/settings/reload` — files are the only write path). It starts nothing itself and fails fast if either URL isn't up. It also prints the active Node/undici version, warning when the local Node major differs from `.nvmrc` — a runtime-specific transport bug is otherwise indistinguishable from a code failure (see [#440](https://github.com/Wave-RF/WaveHouse/issues/440)). - `tests/e2e/sdk/helpers.ts` — JWT factories, typed client constructors, async wait helpers, direct ClickHouse query helper. **Running E2E tests**: ```bash # Build the cover binary, install deps, run all E2E tests make test-e2e ``` `make test-e2e` builds `bin/wavehouse-cov` (coverage-instrumented) and runs the orchestrator under `scripts/orchestrator/` to wire ClickHouse + the cover binary into the suite. covdata flushes on SIGINT into `tmp/coverage/e2e/data/`. The orchestrator always provisions its own stack — fresh ClickHouse and Redis testcontainers plus `wavehouse-cov` on a random free port — so a running `make dev` on `:8080` is neither detected nor reused, and the two don't collide. To run vitest against a stack you manage yourself, start a Redis for the fixture's `cache.backend: redis`, then the server from the **repo root** with the E2E fixture config: ```bash docker compose -f deployments/compose/dependencies.yaml --profile redis up -d WH_CONFIG=tests/e2e/fixtures/config.yaml WH_CACHE_REDIS_ADDRS=localhost:6379 go run ./cmd/wavehouse ``` The fixture matters: the suite signs its tokens with its `sdk-dev-secret` and depends on its dedupe, DLQ, and 5s schema-refresh settings. Point the suite at a default `make dev` server (whose `./settings/config.json` has an empty `auth.jwt_secret`, so the suite's tokens do not validate) and setup's schema calls are rejected, then global setup dies 30s later on a misleading `schema not refreshed within 30s`. The repo root matters too — the fixture's `settings.dir` is relative to the working directory. The fixture's settings directory (policy, pipes, and tunables) points at ClickHouse on `localhost:9000`; if yours isn't there, edit `clickhouse.addr` / `http_port` in `tests/e2e/fixtures/settings/config.json` (the orchestrator patches them itself for its testcontainer). Prefixing the variable to `make dev` does **not** work: that recipe pins `WH_CONFIG=.config.local.yaml` inline, which overrides anything inherited from the environment. Then set `CLICKHOUSE_URL` / `WAVEHOUSE_URL` and run `pnpm test` from `tests/e2e/sdk/`; teardown is a no-op on that path, so your stack survives between iterations. If a previous run was killed (harness timeout, stop button, `SIGKILL`), it can leave a `wavehouse-cov` behind. That process shares `tmp/data` and `tmp/wavehouse-cov.log` with the next run and will corrupt it, so the orchestrator kills any leftover before starting and says so. **Environment knobs**: | Variable | Effect | |----------|--------| | `V=1` | Stream the WaveHouse subprocess log live *in addition to* capturing it to `tmp/wavehouse-cov.log`. The on-failure log excerpt is then skipped — you have already seen it | | `E2E_CH_QUERY_TIMEOUT_MS` | Per-request ceiling for the suite's direct ClickHouse queries (default `10000`) | | `E2E_NO_COVERAGE=1` | Drop `--coverage` from the vitest run (skips v8 instrumentation and report generation) while chasing a flake. **Local debugging only** — no report is written. Ignored (with a log line) under `make ci` / `make test-all`, so an exported-and-forgotten var can't produce a green coverage gate with the TS e2e report missing | **Test files** (`tests/e2e/sdk/*.test.ts`): `admin`, `auth`, `batching`, `cache`, `dlq`, `ingest`, `ndjson`, `query`, `streaming`, `stress`, plus `helpers` — a stack-free unit test of the harness's own `waitForCondition` poll helper rather than a pipeline test. ## Linting ```bash make lint ``` `golangci-lint` is kept out of `go.mod` (its dependency tree conflicts with the main module), so the Makefile pins the version instead and downloads it into `.bin/` on first use — `make lint` and `make tools` both handle it, and no global install is needed. A global copy is never what `make lint` runs, so install one only if you want to invoke `golangci-lint` outside `make`: - **macOS**: `brew install golangci-lint` - **Binary**: See [golangci-lint.run/welcome/install/](https://golangci-lint.run/welcome/install/) The Makefile exports `GOTOOLCHAIN` pinned to `go.mod`'s `toolchain` line if it has one, else its `go` directive, so local runs and CI use the same Go even when your installed Go is newer (the first run downloads that toolchain once). The source-build `deployments/Dockerfile` pins the matching `golang` image tag instead (the image sets `GOTOOLCHAIN=local`), and `make check-dockerfile-go`, part of `make verify`, fails if the tag and `go.mod`'s Go version differ. A bare `go test` outside `make` is not pinned. Raising the `go` directive to a new minor version (1.27, not a 1.26 patch) also needs a golangci-lint release built with that minor or newer: otherwise `make lint-go` refuses with `can't load config: the Go language version (go1.26) used to build golangci-lint is lower than the targeted Go version (1.27.1)`. Bump the directive with `go get go@1.N.P`, because `go mod edit -go=1.27` writes a version `make` cannot pin. The configuration is in `.golangci.yml` (v2 format with `default: none` for explicit control) — that file is the authoritative list of enabled linters. Highlights: - **errcheck** — Unchecked error returns - **govet** — Suspicious constructs - **staticcheck** — Static analysis - **unused** — Unused code - **gosec** — Security issues - **gocritic** — Opinionated style checks - **revive** — Extensible linter (replaces golint) - **ineffassign** — Ineffective assignments - **misspell** — Spelling errors in comments/strings - **bodyclose** — Unclosed HTTP response bodies - **noctx** — HTTP requests without context - **errorlint** — Proper error wrapping checks (`%w`, `errors.Is/As`) - **tparallel** — Missing `t.Parallel()` in test subtests - **depguard** — Import boundaries: only `internal/mq` may import NATS/JetStream (`github.com/nats-io/…`); everything else goes through an mq-owned type. And the `pkg/` packages import only the standard library, each other, `go-humanize` and `yaml.v3` (plus `testify` in tests): an allow list, so a new dependency there fails lint until it is listed Formatting (**gofumpt** — strict superset of gofmt — and **goimports** import grouping) is enforced through the v2 `formatters:` section rather than as linters. ### Markdown and MDX `make lint` also covers documentation: **markdownlint** owns Markdown *and* MDX style (rules in `.markdownlint.json`; file selection in `.markdownlint-cli2.jsonc` for `.md`, plus the `**/*.mdx` glob on `lint:md` in `package.json`) and **misspell** owns spelling. Two rules are repo-local, in `scripts/markdownlint-rules/`: - **WH001 / no-hard-wrapped-prose** — a paragraph must be one line. Hard-wrapping at 72/80 columns turns a one-word edit into a five-line diff. Autofixes. Tables (leading pipes optional), code, headings, setext underlines, blockquotes, JSX, `$$` display math, and multi-line MDX `import`/`export` are left alone; a list item is joined as a unit; an aside's body is joined but its `:::` delimiters are not. Note the four-space rule cuts both ways: any line indented four or more spaces is read as an indented code block and skipped, so a *nested* list item is never joined and must be unwrapped by hand. - **WH002 / mdx-fence-needs-blank-line** — an MDX code fence directly against a JSX tag (`` immediately followed by a fence). MDX renders that fine; the problem is that markdownlint parses CommonMark, where the tag opens an HTML block that runs to the next blank line — so the fence is not a code block to any generic rule, and `markdownlint --fix` will happily reformat the code inside it. The blank line is what keeps the two parsers agreeing. Run `make fix` to apply them. **The generic markdownlint fixers run over `.md` only.** markdownlint parses CommonMark and MDX does not, and where the two disagree an autofix rewrites the inside of a code block — de-indenting YAML comments it reads as headings, autolinking bare URLs. Reporting that disagreement is useful, so `make lint` still checks `.mdx`; acting on it is not. MDX therefore gets exactly one *structural* fixer, `scripts/fix-mdx-fences.mjs`, which only ever inserts a blank line beside a JSX tag. (misspell still auto-corrects spelling in `.mdx` — its corrections come from a curated list and don't depend on parsing the document.) The practical consequence: **a problem `make lint` reports in an `.mdx` file may need fixing by hand**, WH001's hard-wrapped prose included. A bare `markdownlint-cli2 --fix` can't reach MDX either: `.markdownlint-cli2.jsonc` globs `.md` only and the `.mdx` glob sits on `lint:md`, so the guard is structural rather than a rule you have to remember. Improving this — a formatter that understands MDX rather than one that guesses — is tracked in [#499](https://github.com/Wave-RF/WaveHouse/issues/499). The markdownlint extension reads `.markdownlint-cli2.jsonc`, `customRules` and all, so no editor setting is needed — but it only activates on Markdown, so in-editor squiggles cover WH001 in `.md` and never WH002 in `.mdx`. Agent-written files get the same treatment at write time from `.claude/hooks/markdown-on-save.sh` — markdownlint in `.md`, the fence pass in `.mdx` — and everyone else gets it from `make fix`. ## Project Structure ```text WaveHouse/ ├── cmd/ # Binary entry points │ └── wavehouse/ # Standalone all-in-one binary ├── internal/ # Private application packages │ ├── api/ # HTTP handlers, router, middleware │ ├── app/ # Process wiring (build every component, run under one errgroup, release in reverse) │ ├── auth/ # JWT/JWKS authentication middleware │ ├── cache/ # Query cache: Ristretto L1 + the tenant-led version index; the Redis-compatible shared backend │ ├── chconn/ # ClickHouse pools, one per connection tuple (reconciled on settings reload) │ ├── config/ # YAML + env var configuration │ ├── coord/ # Leases with fencing tokens (in-process Local, RunElected, coordtest suite) │ ├── dedupe/ # Optional deduplication (Reserve/Commit/Release; Pebble or DynamoDB) │ ├── discovery/ # ClickHouse schema introspection + validation │ ├── ingest/ # Batch buffering + DLQ + Active Sweeper + shard claims │ ├── keyenc/ # One escaping for composite keys (NATS subject tokens, cache keys, dedupe keys) │ ├── mq/ # MQ boundary: the only NATS/JetStream importer │ ├── observability/ # OpenTelemetry pipeline (traces/metrics/logs + Prometheus) │ ├── query/ # Structured query AST + SQL builder │ ├── settings/ # Settings directory runtime: adopted snapshot, reload, watcher, WriteSeed │ ├── stream/ # SSE fan-out: Hub, Subscriber queue, Bucket, keepalive wheel │ └── testutil/ # Shared test helpers and mocks (cachetest suite) ├── pkg/ # Public packages: importable at a release tag, no internal/ or runtime dependency │ ├── chsql/ # Shared ClickHouse SQL helpers (quoting + bind-safety) │ ├── pipes/ # Named query pipes (types + parameter binding) │ ├── policy/ # Access control policies (types + evaluation) │ ├── settings/ # Settings model: document types, Finding, the validator (on disk and over bytes), embedded seed │ └── tenant/ # Tenant id: type, grammar, reserved default, request header name ├── tests/ # Integration & E2E tests │ ├── integration/ # Go integration tests (//go:build integration) │ └── e2e/ # E2E suite (SDK specs; `scripts/orchestrator` starts ClickHouse and Redis testcontainers) │ ├── fixtures/ # E2E server config + settings-directory fixtures │ └── sdk/ # E2E specs driven through the TypeScript SDK (Vitest) ├── clients/ # Client SDKs │ └── ts/ # TypeScript SDK (@wavehouse/sdk, pnpm workspace) ├── deployments/ │ ├── compose/ # Docker Compose files (standalone.yaml, dependencies.yaml) │ ├── nats/ # Helm values and JetStream manifests for an external NATS (`mq.backend: nats`) │ ├── Dockerfile # Runtime image │ └── Dockerfile.goreleaser # Release image (built by GoReleaser) ├── scripts/ # E2E orchestrator, cov tool, CI/hook helpers ├── docs/ # Documentation ├── config.yaml # Default configuration file ├── Makefile # Build, test, lint, release targets ├── .golangci.yml # Linter configuration ├── .goreleaser.yaml # Release build configuration └── .air.toml # Hot-reload configuration ``` ## Code Conventions - **Strict Go formatting**: Use `gofumpt` (a stricter superset of `gofmt`, enforced by CI). `make fmt` checks it, as does `make verify` (what CI and the pre-commit hook run); `make fix` applies it. - **Interface-first design**: Core behaviors (`Cache`, `Deduplicator`, `Publisher`, `Subscriber`) are defined as interfaces so implementations can be swapped behind a stable contract. - **Package boundaries**: The `internal/` directory ensures packages are private to this module; `pkg/` is the importable surface — what a settings writer outside this repository validates with — so a package there imports nothing under `internal/` and none of the server's runtime dependencies (the `public-packages` depguard rule). - **Error handling**: Return errors to callers. Use `slog` for structured logging, through the default logger (`slog.InfoContext(ctx, …)` and its siblings) — constructors don't take a `*slog.Logger`; tests silence or capture it with `internal/testutil/logtest`. - **Schema-driven**: ClickHouse is the schema source of truth. WaveHouse discovers and validates against real table schemas. ## Makefile Targets Run `make help` to see all targets. Key ones: | Target | Description | | ------ | ----------- | | `make help` | Show all targets with descriptions (always the source of truth) | | `make tools` | Bootstrap: install pinned tools (`golangci-lint`, `air`, `misspell`, `shellcheck`, `actionlint`), Go modules, pnpm deps, and git hooks (`core.hooksPath` → `.githooks/`) | | **Dev** | | | `make dev` | Hot-reload dev server: ClickHouse via Compose + WaveHouse under air on `:8080` | | `make deps-up` | Start ClickHouse alone (idempotent; blocks until healthy) | | `make deps-down` | Stop ClickHouse (preserves data volume) | | `make deps-logs` | Tail ClickHouse logs | | `make deps-shell` | `clickhouse-client` REPL on the running container | | `make deps-wipe` | Stop ClickHouse AND destroy its data volume (DESTRUCTIVE) | | **Observability** | | | `make obs-aspire` | Prebuilt 0-config o11y UI to show WaveHouse metrics, logs, and traces locally | | `make obs-grafana` | Grafana alternative to aspire, more advanced and complicated | | `make obs-front` | Custom graphs like grafana, but is simpler and easier to configure like aspire | | **Static checks** | | | `make fmt` | Check formatting across Go (`gofumpt`) + TS (Biome). Run `make fix` to apply. | | `make tidy` | Verify `go.mod`/`go.sum` are tidy (run `make fix` to apply) | | `make lint` | Run linters across Go (`golangci-lint`) + TS (Biome) + Markdown/MDX (markdownlint) + prose (misspell) | | `make vulncheck` | Run `govulncheck -scan package` (`V=1`: the default symbol-level scan, with example traces) | | `make verify` | Repo-wide static checks: Go (tidy + fmt + vulncheck + lint) + TS (Biome + `tsc` typecheck) + Markdown/MDX (markdownlint + rule fixtures) + prose (misspell) + shell (shellcheck) + workflows (actionlint) + path-classifier fixtures + release-channel fixtures + go-toolchain fixtures + Dockerfile Go-version check + NATS server-line check (`go.mod`, the external-NATS test image, the Helm tag, `recommendedNATSMinor` and `deployment.md` agree) + tagged-test selector and integration-parts fixtures + PR build-cache and `main` cache prune fixtures + test-image prefetch fixtures (`scripts/ci/prefetch-test-images.sh` against the real sources) + review-gate hook tests + git-hook tests (`.githooks/`) + Claude Code hook-command tests (`.claude/settings.json`) + `make ci-remote` tests + docs type-check (`astro check` — not a full build, so link validation is left to `make build-docs`, which `make ci` runs); a bare `make verify` runs these in parallel | | `make fix` | Auto-fixes across Go (`tidy` + `gofumpt` + `goimports` + `lint --fix`), TS (Biome `--write`), Markdown (markdownlint `--fix`), MDX (`fix-mdx-fences` only — the generic fixers never run over `.mdx`), and docs-prose spelling (misspell, both) | | **Build** | | | `make build` | Compile `wavehouse` → `bin/wavehouse` (debug symbols kept) | | `make build-release` | Stripped release-style build → `bin/wavehouse-release` | | `make build-cover` | Coverage-instrumented build → `bin/wavehouse-cov` (used by E2E) | | `make build-ts` | Build TypeScript SDK → `clients/ts/dist/`, then smoke-load its ESM, CJS and IIFE entry points at the oldest Node `engines.node` admits, and type-check a CommonJS and an ESM consumer against the built declarations with `tsc --module node16` | | **Test** | | | `make test` | Alias for `test-unit` | | `make test-unit` | Go unit tests + render coverage + gate suite threshold | | `make test-integration` | Go integration tests (requires Docker) + coverage gate: the parts `test-integration-app` and `test-integration-backends`, at once | | `make test-ts` | SDK vitest unit tests + v8 coverage + gate against `suites.ts-unit` (matches Go's "always coverage" pattern) | | `make cov` | Merge Go + TS coverage and gate against thresholds. Auto-runs after `make test-all` and `make ci`; standalone `make cov` is "show me the merged numbers without re-running." Each side skips silently if its data is missing, but `make cov` fails if *both* are empty (you ran it before any test target). | | `make test-e2e` | E2E SDK suite against `bin/wavehouse-cov` + coverage gate | | `make test-all` | All four suites sequentially + merged coverage gate | | `make ci` | Full pipeline: parallel `verify` + builds + unit/SDK tests, then integration + E2E + cov | | `make ci-remote HOST=` | `make ci` for `HEAD` on an ssh host or `docker:`; writes the same marker if it passes there for this exact tree ([details](#running-make-ci-on-another-machine)) | | **Release** (see [Cutting a release](#cutting-a-release)) | | | `make release-server VERSION=X.Y.Z` | Tag a server release — binaries + container image | | `make release-sdk-ts VERSION=X.Y.Z` | Tag a TypeScript SDK release — npm | | `make release-sdk-go VERSION=X.Y.Z` | Tag a Go SDK release — `go get` (pending [#434](https://github.com/Wave-RF/WaveHouse/pull/434)) | | **Analysis** (informational, not in CI) | | | `make size` | Binary size analysis → `tmp/analysis/` (text + SVG + interactive HTML) | | `make audit-cgo` | Audit dependency tree for C files (builds use `CGO_ENABLED=0`) | | `make deadcode` | Find unreachable functions | | `make dep-cut` | Top cuttable deps by transitive weight (`LIMIT=N` to override) | | `make binary-analysis` | Combined: `size` + `audit-cgo` + `deadcode` | | **Cleanup** (tiered — compose explicitly for partial resets) | | | `make clean` | Build outputs only (`bin/`, `dist/`, `clients/ts/dist/`, `docs/dist/`, `docs/.dev-dist/`) | | `make clean-test` | Test outputs only (`tmp/` — coverage data, logs, NATS state) | | `make clean-tools` | Installed tools and pnpm deps (`.bin/`, `node_modules/`) | | `make clean-all` | Full reset: above + `data/` + Docker volumes | All test targets accept `ARGS="..."` for pass-through `go test` flags. Build targets accept `TAGS="..."` for Go build tags. `V=1` switches to verbose `gotestsum` output. ## Dependency Management ### Updating Dependencies pnpm's hoisted fallback is off (`hoist: false` in `pnpm-workspace.yaml`), so after adding or bumping an npm dependency, one that imports something it doesn't declare fails with `Cannot find package 'X' imported from …/node_modules/.pnpm/…` (ESM) or `Cannot find module 'X'` plus a require stack (CJS) when the importing module loads (in a build, test or dev server), not at install time; an undeclared type-only import isn't caught at all, since with `skipLibCheck` on it silently degrades to `any`. Fix a runtime one with a `packageExtensions` entry in `pnpm-workspace.yaml`: a dependency entry for a plain missing package, a peer entry when the package must share the app's instance (`astro`, for an Astro integration). Report the missing declaration upstream too, so the entry can go: `starlight-image-zoom` needed an `astro` peer entry until 0.16.1 declared it. For Go modules: ```bash go get -u ./... # Update the main module's non-test deps (direct and indirect); add -t for test-only deps, and run go get -u tool for the tool block go mod tidy # Remove unused, add missing ``` ### Vulnerability Scanning `make vulncheck` runs `govulncheck -scan package`: it reports every known vulnerability in a package your code imports, without checking whether the vulnerable function is reachable. That is stricter than govulncheck's default call-graph scan, so a hit does not on its own mean the code path is used. ```bash make vulncheck # package-level scan (what make verify and CI run) V=1 make vulncheck # default symbol-level (call-graph) scan, with example call traces ``` For a combined security scan, run `make verify` — it runs `vulncheck` alongside `lint`, and `gosec` is one of the linters enabled in `.golangci.yml`. This is also what CI runs on every push and pull request. ### Dependabot Dependabot is configured in `.github/dependabot.yml` to open weekly grouped PRs for three update configs: - **Go modules** (root) — outdated or vulnerable Go dependencies, commit prefix `deps:`. - **GitHub Actions** (root **and** `/.github/actions/setup-env`) — outdated action versions tracked against the SHA pins across `.github/workflows/*` and the `setup-env` composite action, commit prefix `ci:` - **npm — pnpm workspace** (root) — covers all three TypeScript packages (the docs site, the SDK, and the E2E tests) in one grouped PR, commit prefix `deps:` PRs are grouped per config to reduce noise. The npm config is pointed at the workspace **root** (`directory: /`), not the individual member directories. The repo has a single root `pnpm-lock.yaml`, and Dependabot only updates a lockfile co-located with the manifest it targets — so a per-member config (the previous setup) bumped a member's `package.json` without regenerating the root lockfile, and every such PR then failed CI's `pnpm install --frozen-lockfile` with `ERR_PNPM_OUTDATED_LOCKFILE`. Pointing at the root lets Dependabot read `pnpm-workspace.yaml`, walk every member, and update the one lockfile. The GitHub Actions config names **two** directories. `directory: /` reaches `.github/workflows/` but does not descend into `.github/actions/*/action.yml`, so the `setup-env` composite action — which owns every cache in `ci.yml` — was invisible to Dependabot, and its pins went stale against upstream and diverged from `publish-npm.yml`, which Dependabot *does* track and which doesn't call `setup-env`. Listing its directory under `directories:` brings it into the same weekly group; **adding a composite action means adding its directory there**, because nothing else catches the drift. The npm config sets a 7-day `cooldown` to match pnpm's `minimumReleaseAge`, with the same `@wave-rf/*` and `@wavehouse/*` exclusions as `minimumReleaseAgeExclude`. It covers only the package Dependabot bumps, so transitive packages can still fail pnpm's check ([#441](https://github.com/Wave-RF/WaveHouse/issues/441)), and security updates bypass it. That remainder fails loudly rather than silently: pnpm 11.21 checks every lockfile entry against `minimumReleaseAge` even under `--frozen-lockfile`, so a Dependabot lockfile that pulled in a too-young transitive package fails the `pnpm install` inside `make verify` with `ERR_PNPM_MINIMUM_RELEASE_AGE_VIOLATION` and nothing merges. Dependabot cannot repair it itself, so a human does. Commit the fix to the Dependabot PR's own `dependabot/npm_and_yarn/**` branch, the same branch an automated write-back would target. Set the manifest range of the bumped package to a release that resolves under the policy, typically the newest release at least 7 days old, which may be newer or older than the version the PR proposed (the PR's own version can keep the young transitive pins). Then run plain `pnpm install` so pnpm resolves only the new subtree under the policy and leaves every untouched pin alone, and run `pnpm install --frozen-lockfile` to confirm the result is policy-clean. On PR #571 the working fix moved `astro` from `^7.1.1` to `^7.2.10`, not to the PR's 7.2.8. `pnpm clean --lockfile && pnpm install` also works but re-resolves the whole tree and sweeps in unrelated bumps. Automating this with a workflow that regenerates the lockfile and pushes it back to the `dependabot/npm_and_yarn/**` branch needs a token that can push and trigger CI, which is not set up; that remains the open part of [#441](https://github.com/Wave-RF/WaveHouse/issues/441). `typescript` majors are held back (`ignore: version-update:semver-major`) because `tsup` vendors a `rollup-plugin-dts` that crashes on TypeScript 7 during `clients/ts`'s `prepare` script — i.e. inside `pnpm install`, which takes every Node job down at once. See the comment in `.github/dependabot.yml` for the condition that lets it be removed. **No auto-merge.** Dependabot PRs go through the same merge gate as any other PR — an approval from the `@Wave-RF/wavehouse-admins` team (the ruleset's `required_reviewers` rule) plus the required checks. (The former `dependabot-automerge.yml`, which auto-approved and merged patch/minor bumps hands-off, was removed — every bump now gets a human admin review.) ## Cutting a release Cutting a release is **one tag** — no version bump in code, no release branch: ```bash make release-server VERSION=0.1.0 # tag v0.1.0 → binaries + container image make release-sdk-ts VERSION=0.1.0 # tag clients/ts/v0.1.0 → @wavehouse/sdk on npm ``` `make release-sdk-go` exists too, wired ahead of the Go SDK landing ([#434](https://github.com/Wave-RF/WaveHouse/pull/434)); it refuses to run until `clients/go/` is in the repo. The one thing to do *before* tagging is promote the changelog: `AGENTS.md` requires every PR to add its entry under `## Unreleased`, so open a PR renaming that heading to `## [X.Y.Z] - YYYY-MM-DD` and adding the matching link reference at the foot of the file. When the release needs upgrade steps, add a `## From X to Y` section to [Upgrading](/upgrading) in the same PR. Nothing in the release pipeline reads `CHANGELOG.md` — this is for the file's own readers. **After** tagging an SDK release, raise `docs/package.json`'s `@wavehouse/sdk` range to the new line. A caret range on a `0.x` version floats only within that minor (`^0.1.1` stops at `0.2.0`), so a new **minor** does not reach the docs site — including its live hero demo — until someone bumps it, which is the point: the bump is where you decide that the deployed demo backend and the site's SDK move together. `minimumReleaseAgeExclude` already exempts `@wavehouse/*` from pnpm's 7-day `minimumReleaseAge` (and the Dependabot `cooldown` excludes it too), so the freshly published version installs immediately. Dependabot will **not** propose this one for you — `@wavehouse/sdk` is in the npm `ignore` list in `.github/dependabot.yml` for exactly this reason, since a grouped Monday `deps:` PR is reviewed by someone thinking about dependency hygiene, not about which wire the demo backend speaks. Each runs [`scripts/release.sh`](https://github.com/Wave-RF/WaveHouse/blob/main/scripts/release.sh), which preflights (on `main`, clean tree, in sync with `origin/main`, the tag free both locally and on the remote, the required `CI` check green on *this exact commit*), prints exactly what will be published, and asks before pushing. `DRY_RUN=1 make release-…` stops after the plan. Tag creation is admin-only via the `release tag protection` ruleset. The components are independent — releasing the server does not publish the SDK, and their version numbers need not match. ### Why there is no version to bump The **tag is the version**, everywhere: | Component | Where the version comes from | | --- | --- | | Server | GoReleaser's `-ldflags` at build time, from the tag | | Go SDK | The tag itself — a Go module has no version file | | TypeScript SDK | `publish-npm.yml` stamps `clients/ts/package.json` from the tag before publishing | The npm case is the one that usually forces a bump commit, and it is deliberately not the source of truth here: the `main` ruleset forbids pushing to `main` directly, so bumping `package.json` would mean a reviewed PR merged before *every* release. The committed value is documentation — if it drifts from the tag, the workflow logs a notice and the tag wins. ### Tag naming ```text v0.1.0 server (root Go module) clients/ts/v0.1.0 @wavehouse/sdk clients/go/v0.1.0 Go SDK ``` The `clients//` prefix isn't a style choice. Go resolves a module in a subdirectory only against a tag carrying that subdirectory as a prefix ([the module reference](https://go.dev/ref/mod) is explicit), so `go get github.com/Wave-RF/WaveHouse/clients/go@v0.1.0` requires precisely `clients/go/v0.1.0`. Every client follows that shape rather than leaving Go as the exception, and a new language is one more `clients//v*` family. Tag globs are anchored at the start of the ref name, so `v*` never matches a `clients/...` tag — which is what keeps a client release from triggering a server release. ### What a release publishes - **Server —** a **GitHub Release** with the Linux archives (amd64 and arm64, `.tar.gz`) and `checksums.txt`. A tag carrying a prerelease suffix (`v0.1.0-alpha.1`) is marked as a GitHub pre-release, so it never takes the "Latest release" badge from a shipped stable version. - **Both —** **release notes generated by GitHub** from the PRs merged since the previous tag *in the same family* — one line per PR, since `main` is squash-merged, grouped into the categories defined in [`.github/release.yml`](https://github.com/Wave-RF/WaveHouse/blob/main/.github/release.yml). Grouping is by **PR label**: `github_actions` / `documentation` are applied automatically by `actions/labeler`, but `breaking-change`, `security`, `bug`, and `enhancement` are applied by hand — an unlabelled PR lands in "Other changes". Dependabot is split out by **author** rather than by label, because the labels `actions/labeler` applies by path — `github_actions`, `documentation` — mark our own PRs too; our CI work gets its own "CI & build" section — ordered above Documentation, since a CI PR here nearly always updates docs too — and Dependencies is pure Dependabot residue. **Any category keyed on a label a Dependabot PR can carry needs that author exclude** — labeler's path labels *and* the ecosystem labels Dependabot applies itself (`dependencies`, `javascript`, `go`, `github_actions`; `javascript` is in neither `labeler.yml` nor our categories) — or that category intercepts bumps before they reach the `📦 Dependencies` catch-all. `CHANGELOG.md` is *not* the source of the release body; it is the longer-form record of why each change was made. - **Server —** a **GHCR image** at `ghcr.io/wave-rf/wavehouse`, with two tags: the immutable `:vX.Y.Z`, and one moving *channel* pointer. A stable release moves `:latest`; a prerelease moves `:alpha` / `:beta` / `:rc` / `:next` instead, matching the npm dist-tag it would get. The channel comes from the **first** prerelease identifier, matched **exactly**: `v0.2.0-rc.1` → `:rc`, while `-alpha1`, `-preview.1`, or any other form → `:next`. `scripts/ci/release-channel.sh` is the single rule every publisher uses, so `ghcr.io/wave-rf/wavehouse:rc` and `@wavehouse/sdk@rc` can't drift apart. **A prerelease-only project therefore has no `:latest` tag** — that is deliberate; `:latest` starts existing when the first stable release ships. - **TypeScript SDK —** an **npm publish** of `@wavehouse/sdk` under `latest` (stable) or `alpha`/`beta`/`rc`/`next` (prerelease), plus its own GitHub Release. - **Server —** **build-provenance attestations** (Sigstore, free for public repos) over every archive and over the image's multi-arch manifest digest; the image attestation is stored alongside the image in GHCR. The release job verifies its own attestations before finishing, so a release that publishes unverifiable provenance goes red. - **TypeScript SDK —** an **npm provenance attestation** via `npm publish --provenance`, surfaced as the provenance badge on the package page and checkable with `npm audit signatures`. The npm job does not re-verify it the way the server job does. Binaries installed with `go install github.com/Wave-RF/WaveHouse/cmd/wavehouse@vX.Y.Z` get no `-ldflags`, so they read their version from the module metadata the Go toolchain embeds; `/version` reports the tag without its leading `v` (`0.1.0`, not `v0.1.0`), with `git_commit`/`build_time` as `unknown`. Release archives and container images carry all three. Consumers can check provenance for themselves: ```bash gh attestation verify wavehouse_linux_amd64.tar.gz \ --repo Wave-RF/WaveHouse \ --signer-workflow Wave-RF/WaveHouse/.github/workflows/release.yml gh attestation verify oci://ghcr.io/wave-rf/wavehouse:v0.1.0 \ --repo Wave-RF/WaveHouse \ --signer-workflow Wave-RF/WaveHouse/.github/workflows/release.yml ``` `--signer-workflow` is not optional garnish: `--repo` alone accepts an attestation produced by *any* workflow in the repo. Same point, and the `:dev` equivalent, in [Deployment → Registry](/deployment#registry) and `SECURITY.md`. :::note[Releasing from the GitHub UI instead] Publishing a release from **Releases → Draft a new release** creates the tag, which fires the same workflow — so it works, and GoReleaser's default `mode: keep-existing` (the key is not set in `.goreleaser.yaml`) means it will not overwrite notes you wrote. Two things the `make` targets do for you and the UI does not: none of the preflight checks run, and you must click **Generate release notes** yourself, because a body you publish empty stays empty. ::: ## The `dev` channel Between releases, every push to `main` republishes both artifacts so `@dev` always means "current `main`": - **`ghcr.io/wave-rf/wavehouse:dev`** — a rolling pointer, plus an immutable `:dev-` (pruned after 30 days by `cleanup-ghcr.yml`, newest 5 always kept). Built by the same GoReleaser pipeline with `WAVEHOUSE_DEV=1`, which suppresses the GitHub Release. Note a Docker tag is only a pointer: `docker run …:dev` reuses a stale local image unless you `docker pull` first or pass `--pull=always`. - **`@wavehouse/sdk@dev`** — `0.0.1-dev..h`, published only when the published package actually changes. The trailing hash covers every file `npm pack` would ship — the built `dist/`, `package.json` minus its `version`, and the bundled `README`/`LICENSE` — so a push whose package would be byte-identical to the current `dev` publish is skipped, while a change to `exports`, `files`, `bin`, or `engines` republishes even though `dist/` is untouched. The `` is load-bearing, not decoration. `npm install …@dev` records a *range* in your `package.json`, not the dist-tag, so what you get on the next install is the highest version matching that range. Under the old `0.0.0-dev.h` scheme, semver's lexical ordering of alphanumeric prerelease identifiers meant the newest publish routinely wasn't the highest one, and a range could resolve *backwards* — an `@dev` install landed on a two-month-old build ([#475](https://github.com/Wave-RF/WaveHouse/issues/475)). A numeric identifier compares numerically, so the channel now orders by publish time. The `0.0.1` base keeps the channel below every real release (so a dev build can never satisfy `^0.1.0`) and above the legacy `0.0.0-dev.*` publishes, which npm's 72-hour unpublish window makes permanent. Docker needs no equivalent fix: a tag is a mutable pointer resolved fresh on every pull, with no version ranges and no ordering, so the failure mode #475 describes cannot arise there. ## CI & review automation AI automation sits alongside the normal CI checks — advisory PR review from CodeRabbit and Copilot, plus the local Claude Code tooling in `.claude/`. Full detail lives in `AGENTS.md` §Repository Automation; this section covers the contributor-facing behavior. ### PR title and Conventional Commits PR titles must match Conventional Commits format and stay ≤ 72 characters — the title becomes the squash-merge commit subject. Both rules are enforced by the `PR title` job under the required `CI` check (`.github/workflows/ci.yml`); validate locally with `scripts/lint-pr-title.sh ""`: ```text <type>(optional-scope)(optional-!): <lowercase subject, no trailing period> ``` Allowed types: `feat`, `fix`, `docs`, `refactor`, `test`, `chore`, `ci`, `deps`, `build`, `perf`, `revert`, `style`. The `!` before `:` marks a breaking change per Conventional Commits 1.0.0 (e.g., `feat!: remove deprecated endpoint`, `refactor(api)!: rename handlers`). Titles are also capped at **72 characters** — they become squash-merge commit subjects (Dependabot PRs are exempt from the cap). If the title doesn't match, a sticky comment posts on the PR explaining the format (from the `PR housekeeping` workflow, which mirrors the same script); it auto-removes once the title is fixed. Fixing the title needs no new push — the edit triggers housekeeping, which re-runs the failed `PR title` job (the job re-reads the title from the API, not the stale event payload). ### Required status checks The `main branch protection` ruleset requires one status check to pass before any PR can merge: - `CI` — the aggregator job of `.github/workflows/ci.yml`. The workflow is a job DAG over the same Makefile targets local `make ci` runs: `lint` (`make verify`), `unit` (`make test-unit test-ts`), `integration` (one job per part of `make test-integration`: `make test-integration-app`, `make test-integration-backends`), `e2e` (`make -j test-e2e` — builds its own SDK dist + cover binary on a warm cache, runs the suite exactly like a local run), `coverage` (`make cov` over every suite's uploaded coverage fragment + threshold gates, like local `make ci`'s final step), `docs-build` (`make build-docs` when docs-affecting files changed, uploading the docs dist artifact), `PR title` (Conventional Commits), and the docs preview/deploy jobs. The aggregator fails if any job failed or was canceled and treats skipped jobs as passing — docs-only PRs skip the Go test suites by design, and fork PRs run everything except the (secret-bearing) docs deploys. Every run's Summary page gets a per-job wall-clock table from the non-gating `Timing summary` job. The full architecture — DAG diagram, design invariants, cache policy, how to add a job — lives in [`.github/workflows/README.md`](https://github.com/Wave-RF/WaveHouse/blob/main/.github/workflows/README.md). The `PR housekeeping` workflow still runs on every PR (labels + the title explainer comment) but is no longer a required check. A separate `Cache cleanup` workflow deletes a closed PR's Actions caches, after each PR run deletes the Go build cache entries no run will read, and after each `main` run deletes `main`'s superseded cache generations. The ruleset also requires an approval from the `@Wave-RF/wavehouse-admins` team (the `required_reviewers` rule — this is what mandates an admin sign-off, replacing the old `Admin approval` status-check workflow), plus 1 approving review, approval of the most recent push by someone other than its author, resolution of all review threads, linear history, no branch deletion, no force-push, and squash-merge only. Repository admins may bypass these requirements when merging their own PR (e.g. a trivial `.github` change) but still cannot push directly to `main`. Approved, green PRs land through a **merge queue** ("Merge when ready"): the queue re-runs the required `CI` check against the PR merged with *current* main (a `merge_group` event — the CI workflow runs the full test suite for these) and fast-forwards only on green. That integration re-test replaces the old "branch is out-of-date with the base branch" requirement — queued PRs don't need manual branch updates, and the queue never pushes to the PR branch. Dependabot PRs go through the same admin review as any other PR — there is no auto-merge (see the Dependabot section above). ### Merge behavior Squash-only merges. The **PR title** becomes the commit subject (with `(#NN)` appended automatically), the **PR body** becomes the commit message. Keep PR bodies tight — they land in `git log` on `main`. The PR template gives the right shape (Summary / Test plan / Related Issues). Include `Closes #NN` in the PR body to auto-close the related issue on merge. Alternatively, link the issue in the sidebar's **Development** section — that triggers auto-close even without the keyword. Auto-merge is enabled repo-wide: click "Enable auto-merge (squash)" on a PR and it merges once checks + approvals land. ### AI reviewers Advisory PR review comes from marketplace apps configured at the org/repo level: - **CodeRabbit** — automated PR review; auto-reviews on open + push, re-trigger with `@coderabbitai review`. - **Copilot** — tied to individual reviewer subscriptions; shows up on PRs where a maintainer with Copilot Pro is listed as a reviewer. Both are **advisory** — the ruleset's `required_reviewers` rule (an `@Wave-RF/wavehouse-admins` approval) plus its thread-resolution / linear-history / required-check rules are the actual merge-gate. ### Reviewer assignment and the Task Board Reviewer assignment is GitHub-native, and so is the board: - **Reviewer assignment**: the `main branch protection` ruleset's `required_reviewers` rule requests the `@Wave-RF/wavehouse-admins` team on every PR, and the team's **code-review assignment** (configured on the team) auto-assigns and load-balances a specific member. No workflow is involved; `dismiss_stale_reviews_on_push` clears approvals on new commits, and GitHub re-requests per its own rules. - **Merge gate**: the ruleset's `required_reviewers` rule requires an `APPROVED` review from the `@Wave-RF/wavehouse-admins` team; it also adds review-thread resolution, linear history, and squash-only. Auto-merge (squash) takes over once checks and approvals land. - **Task Board** (Projects v2, project #7): card placement and status are handled by GitHub-native Projects v2 automation configured in the project UI — there is no workflow-driven board state machine. Priority lives on the board's `Priority` field (set during issue triage, below). Dependabot PRs go through the same admin review as any other PR — there is no auto-merge (see the Dependabot section above). ### Invoking bots manually - **CodeRabbit**: comment `@coderabbitai review` to re-trigger a review, or `@coderabbitai <question>` to ask it something. Works in both top-level and inline review comments. - **Copilot**: the re-request-review button on the PR page sends a fresh request. ### Review-response expectations Every review comment (human or AI) must get a substantive reply before merge — not "fixed" alone. The ruleset's `required_review_thread_resolution: true` means unresolved conversations literally block merge. Agents working on PRs follow the pattern documented in `AGENTS.md` §"Review Response": accept / push back / defer, reply with detail, resolve when settled. When pushing back on a bot's suggestion, end the reply with the bot's mention (e.g. `@coderabbitai`) to invite a counter-reply so the dialog actually loops. ### Issue triage Issue triage is **manual**. There is no triage workflow: `.github/workflows/triage.yml` classified new and edited issues with GitHub Models, which GitHub [retired on 2026-07-30](https://github.blog/changelog/2026-07-30-github-models-is-now-retired/) — the inference endpoint returns `410` for every request, so the workflow failed on every issue open/edit until it was removed ([#431](https://github.com/Wave-RF/WaveHouse/issues/431)). What that means in practice: - `area/*`, `security`, and `breaking-change` labels are applied by hand when an issue is triaged. PR labels are unaffected — those come from `actions/labeler` (below), which is path-based and needs no model. - Priority on the **Task Board** project #7 is set by hand on the board's `Priority` field. `.github/board-config.env` existed only for that workflow and is gone with it. The `PROJECT_BOARD_TOKEN` repo secret is now **unused** — it has no consumers left, so it is worth deleting and revoking the PAT behind it. - Maintainers running Claude Code can use the `/pm-triage` skill, which does the same classification locally against the live board. ### Auto-labeling PRs The `PR housekeeping` workflow (`.github/workflows/housekeeping.yml`) runs `actions/labeler` with `.github/labeler.yml` to apply `area/*`, `dependencies`, `github_actions`, `go`, and `documentation` labels to PRs based on the files they change. Sync-mode: labels follow the current changed-file set. ### When adding a new `internal/<pkg>/` package Follow the checklist in `AGENTS.md` §"Common Tasks / Adding a new internal package" — the automation-relevant steps are: 1. Create a matching `area/<pkg>` repo label with a meaningful description. 2. Add the path → label mapping to `.github/labeler.yml` so PRs touching the new package get auto-labeled. Step 2 is the one that automates anything — PR labeling is path-based. Step 1's label is applied by hand during issue triage.