typesafe-mcp

evaluate is a stdio MCP server for TypeSafe Jev. Coding agents send observed state plus typed questions and get back probabilities, choices, and scores they can branch on in code.

Jev is a decision primitive. The host still reasons, edits, and executes.

Install | Setup | The tool | Runtime | Quality gates | Contributing

┌────────────────┐   evaluate tool    ┌──────────┐   POST /v1/systemone   ┌──────────────┐
│ Claude Code    │ ─────────────────▶ │ evaluate │ ─────────────────────▶ │ TypeSafe API │
│ Claude Desktop │   MCP stdio        │  binary  │     or /decisions      │  or          │
│ Codex          │                    │          │     retries 429 / 529  │  OpenRouter  │
│ pi · OMP       │ ◀───────────────── │          │ ◀───────────────────── │              │
│ PyThinker · …  │   typed JSON       │          │                        │              │
└────────────────┘                    └──────────┘                        └──────────────┘

The tool contract is canonical in this binary: description, field docs, and agent guidance live in cmd/evaluate/tools.go. Claude Code, Claude Desktop, and Codex register that MCP surface. Stock pi has no MCP client, so evaluate setup pi writes an extension that registers the same tool and talks MCP to this binary. Pi that already loads MCP servers (pi-mcp-adapter and similar) registers evaluate mcp in that adapter's config, same as any other stdio server. Adapters own transport and registration only; they do not author Jev semantics.

Install

macOS and Linux, amd64 and arm64:

curl -fsSL https://raw.githubusercontent.com/PyModel/typesafe-mcp/main/install.sh | sh

Installs to ~/.local/bin. Add it to PATH if needed: export PATH="$HOME/.local/bin:$PATH". With Go 1.27+:

go install github.com/PyModel/typesafe-mcp/cmd/evaluate@latest

evaluate update upgrades in place from a checksum-verified GitHub release.

Setup

Get a key at https://console.typesafe.ai/

TYPESAFE_API_KEY=your-key evaluate setup

That finds the agents installed on this machine, lists them, and asks which to register with — Enter takes all of them. --yes skips the question; --host omp --host opencode names hosts outright. Installing the binary does not register it: run this once after install.sh.

Config files are edited in place: the evaluate entry is inserted or replaced, and everything else — comments in the JSONC ones included — stays as written. A file that does not parse is refused, not rewritten. DSH composes its config from patch layers keyed by entry id, so its registration is one entry in the home layer, which applies over every profile; a patch file this setup did not write is left alone and reported, since merging it would mean parsing DSH's whole YAML dialect. Aider has no MCP client; Warp keeps MCP servers inside the app (Settings → AI → MCP).

evaluate setup mcp is the older, narrower form: Claude Code (user scope) and Codex when their CLIs are on PATH, and Claude Desktop when the app is installed. Every TYPESAFE_* variable in the shell is copied into the client config, plus OPENROUTER_API_KEY. Re-run to refresh.

Already on OpenRouter? Use that key and evaluate calls OpenRouter's Decisions endpoint, billed there:

OPENROUTER_API_KEY=your-key evaluate setup mcp

TYPESAFE_API_KEY wins if both are set, so a leftover OpenRouter key cannot reroute an existing TypeSafe setup. OpenRouter's Decisions path is still /api/alpha/ and may move.

For stock pi:

evaluate setup pi

Writes ~/.pi/agent/extensions/evaluate.ts (or $PI_CODING_AGENT_DIR/extensions/). Run /reload in pi. The extension reads the API key from the shell pi runs in; nothing is baked into the file.

If pi already loads MCP servers (pi-mcp-adapter and similar), add this binary to the adapter config (~/.pi/agent/mcp.json, or .mcp.json in the project):

{
  "mcpServers": {
    "evaluate": {
      "command": "/absolute/path/to/evaluate",
      "args": ["mcp"]
    }
  }
}

Reload pi. The adapter inherits TYPESAFE_API_KEY or OPENROUTER_API_KEY from the shell pi runs in, unless you set env on the server. Use one registration: the extension and an mcp.json entry both expose a tool named evaluate.

Any other MCP-capable agent or CLI: register the same stdio server by hand — command /absolute/path/to/evaluate, args ["mcp"], with TYPESAFE_API_KEY or OPENROUTER_API_KEY in its env — using the mcpServers entry above in whatever file that host reads. The tool, its schema, and its guidance are identical on every host. Details: cmd/evaluate/AGENTS.md.

With the TypeSafe skill

TypeSafe ships an official agent skill that teaches a coding agent the question types, patterns, and cookbooks. The two are complementary, not alternatives:

# Claude Code
claude plugin marketplace add typesafe-ai/skills
claude plugin install typesafe@typesafe-ai

# other agents
npx skills add typesafe-ai/skills --skill typesafe-ai

evaluate does not depend on the skill: its tool description and server instructions carry the guidance needed to call it correctly.

The tool

Interactive map with themes, path tracing, and a relationship lens: docs/architecture.html.

One tool, evaluate. Read-only. Request:

Malformed noul / choice / score criteria are rejected locally, with questions["id"].criteria in the error, before anything is sent. The documented answer space is enforced locally too: choice takes 1–255 options and score 1–10 levels, and anything outside that fails before a byte reaches the API. Unknown question types pass through; the API documents more types than this tool.

Successful responses are validated against your request before they reach you: every question id must be answered exactly once, and each answer is checked against the criteria you sent — distribution keys, score range, probability sums, confidence. A malformed or partial upstream response becomes a tool error naming the offending answer, never silent garbage.

Independent questions over the same state belong in one call. They run in parallel and cannot see each other's answers.

Do not put your conclusion in state. Jev treats that as evidence, and the confidence that comes back is agreement with you, not an independent check.

Example:

Use evaluate to decide whether this ticket is urgent and which team should own it: Help! My payouts have been failing for 3 days.

{
  "state": "Help! My payouts have been failing for 3 days.",
  "questions": {
    "is_urgent": {"type": "noul", "instructions": "Does this convey urgency?"},
    "department": {"type": "choice", "instructions": "Which team should handle this?",
      "criteria": {"billing": "Payments, refunds", "technical": "Bugs, outages", "sales": "Pricing"}}
  }
}

The tool returns the raw TypeSafe JSON, answers keyed by the same ids.

Runtime

• 429 and 529 are retried with exponential backoff (up to 3 retries). Other API errors come back as tool errors.
• HTTP timeout 60s. Response body capped at 16 MiB.
• Safe under parallel calls: each call owns its retry policy, and concurrent requests cannot amplify retries or serialize into each other's backoff windows.
• Single static binary: no Node, no Python runtime for the server.

How to write questions, batch, and read confidence: docs/jev/. Live TypeSafe docs: https://docs.typesafe.ai · https://docs.typesafe.ai/api

Quality gates

The tool's contract is pinned by tests at every layer:

• Request matrix — malformed inputs, criteria shapes, answer-space limits, and model defaults, exercised through the real MCP boundary, with golden error texts and proof that errors never echo your state back.
• Response matrix — every successful Jev response is validated against the questions that produced it (answer ids, criteria match, distributions) before it can cross the tool boundary.
• Real stdio proof — a subprocess suite speaks newline-delimited JSON-RPC to the shipped binary: clean flow, hostile-frame behavior, and the stdout-is-protocol-only invariant.
• Concurrency — parallel calls keep aggregate retries bounded and backoff windows overlapped.
• Live gate — task smoke proves the implementation still agrees with the production API (opt-in, needs a key); any change that tightens what the tool accepts or rejects must pass it before landing.

CI runs offline and deterministic; live and model-driven checks are explicit opt-in tasks. Eval plan: docs/evals/PLAN.md · Product roadmap: docs/ROADMAP.md.

Evals

The tool-selection gate (docs/evals/PLAN.md §4–§6) measures whether a coding agent reaches for evaluate exactly when semantic judgment is the right primitive — and not otherwise. A real agent loop (temperature 0, five turns max) works against the shipped binary plus harness-side decoy tools (read_file, run_command, answer_directly); every tool call is graded against a rubric fixed at authoring time over a 20/20/10 corpus, three repetitions per scenario. A naive always-evaluate policy must fail the corpus (vacuity), or the numbers mean nothing.

Three numbers are pre-registered as pass thresholds: tool_precision (appropriate evaluate calls ÷ all evaluate calls), request_validity (schema-accepted calls ÷ attempted calls), and ambiguous_acceptance (ambiguous scenarios whose majority passed). The remaining metrics are reported, not gated. The first gated run — deepseek-v4-flash served locally — is red: precision 0.778 < 0.90 and request_validity 0.815 < 1.00. A second independent run at a later head reproduced every aggregate and the same seven failing scenarios exactly (trend chart above). Per the pre-registered failure attribution, every failure is single-model: six positives answered in prose (missed_call_rate 0.300) and one case where the model looped evaluate five times on schema-rejected arguments. That is a model limitation, not product ergonomics, so no tool-description rewrites are triggered.

Run records live under evals/results/ — append-only, one JSON per run, committed so the history is auditable. Each record keeps every rep's observed tool calls, so a grading fix is re-scored offline instead of costing another model run (ADR-003). Regenerate the charts with task charts after a run; task eval runs the gate itself (opt-in: a local OpenAI-compatible model plus a TypeSafe or OpenRouter key).

Agent working practices

House rules: AGENTS.md. Pack (verification, orchestration, skills, prompts): agents/. Roles: roster/. Workflows: playbooks/. Failure principles: ANTIPATTERNS.md.

Contributing

Maintained by elkaix under PyModel. Issues and PRs: PyModel/typesafe-mcp.

Task: task check (scripts/check.sh) runs gofmt, go vet, and tests with -race in an environment with no keys, as CI does; task hooks installs it as git hooks (lint on commit, the full gate on push). task inspect opens the MCP Inspector against a local build. task smoke checks the live API. Contributor notes: AGENTS.md. Domain terms: docs/CONTEXT.md.

MIT. See LICENSE.