Herdr, T3 Code and Amp all promise to help you run lots of coding agents at once, but they aren’t the same kind of tool. Herdr and T3 Code are control surfaces for agents you already have: Herdr in your terminal, T3 Code as a desktop, web and mobile app. Amp is a coding agent with its own cloud machines and its own manager agent, Puck. To see how they compare in practice, I built the same minimal software factory with each one: open GitHub issues, let agents turn them into pull requests, and have the merges deploy a website to GitHub Pages.

All three got there, on free plans, with my Claude subscription doing the work. Each factory shipped five issues to the live site in under eight minutes with no help from me. None of them noticed new issues on its own. About an hour in, GitHub suspended my account, and I explain why below.

Herdr vs T3 Code vs Amp: the quick verdict

The speed differences are small and mostly reflect how long each Claude run took. What separates the tools is how you watch and steer the agents, and what each one locks behind a paid plan.

What each tool actually is

It’s easy to assume you’d use T3 Code to manage your Amp agents, until you notice Amp has its own manager, Puck. The tools sit in two layers: agents like Claude Code, Codex and Amp hold the model and do the work, and Herdr and T3 Code sit on top of agents you already have so you can run, watch and steer many at once. Amp is the odd one out, because it brings its own control layer too.

• Herdr is a terminal workspace manager for coding agents, written in Rust. It works like tmux, with panes, prefix keys and sessions that survive disconnects, but it also knows which agent is in each pane and whether it’s working, blocked or done. That’s the difference from plain tmux, which only shows you terminals. Its command-line tool lets one agent create worktrees, start other agents, prompt them and wait for them.
• T3 Code, from Theo Browne and Ping.gg, is a desktop, web and mobile app for the agents already installed on your machine, using your existing subscriptions. It isn’t a harness itself: it gives each Claude Code or Codex thread its own git worktree and shows its status and pull request. It also installs a t3 command for starting and messaging threads from a script. It calls itself "very very early", and I used version 0.1.5.
• Amp is a coding agent. Its threads can run on “orbs”, which are Amp’s cloud machines billed by the minute, or on your own computer as a “runner”. Puck, its manager agent, starts and monitors Amp threads from the web, the apps, Slack or any MCP client. Together they do the job of Herdr and T3 Code, but only for Amp.

How I tested

The goal was the smallest software factory that’s still a factory: issues go in, a live website comes out, and I don’t touch the code.

• The site. A static site called Cape Trails, about day hikes around Cape Town, in plain HTML, CSS and JavaScript. The starting code had three trail cards, a planted bug (the Kasteelspoort card opened the Pipe Track page) and a typo in the footer.
• The repo. Each tool’s agent created its own public repository from the same starting folder, turned on GitHub Pages, and confirmed the site was live. A check workflow ran a link checker on every pull request, and every push to main deployed the site.
• The rules. The same AGENTS.md in each repo: one branch per issue, open a pull request that says Closes #N, wait for the check, squash-merge, rebase if the merge conflicts. It also said to touch no other repository.
• Wave 1. I opened five issues at once: add a Lion’s Head trail page, add difficulty filter buttons, add dark mode with a toggle, fix the Kasteelspoort link, and “Homepage feels a bit plain. Can we make the homepage look nicer?” Filters, dark mode and the homepage polish all touch the same CSS, so conflicts were likely.
• The dispatcher. In each tool, one agent got the same prompt: list the open issues, start a separate agent per issue in its own git worktree using this tool, run them in parallel, keep them going until every issue is closed by a merged pull request, then check the live site. The only difference between the three prompts was one line naming the tool’s command.
• Wave 2. After wave 1, I opened two more issues (an About page and the footer typo) and waited ten minutes without saying anything, to see if anything picked them up. Then I told the dispatcher: “Two new issues have been opened. Handle them the same way.”
• Free plans only. Herdr and T3 Code are free. Amp was on the free Hobby plan with my Claude subscription linked, running in Amp’s “Claude Code mode” on my Mac as a runner. Every agent in every tool was Claude Opus 5.5 through the same subscription, so the tools are the only thing that changes.
• No approval prompts. Amp and T3 Code ran agents with full access by default. In Herdr I started Claude Code with --dangerously-skip-permissions to match.
• Timings come from GitHub: when each issue was opened, when its pull request was opened and merged, and when the Pages deploy containing it finished.

I ran the tools one after another, on 30 October 2026, so they weren’t competing for the same subscription limits.

Amp Code: works free, but Puck doesn’t

Amp’s free Hobby plan lets you link a Claude or ChatGPT subscription and run Amp in “Claude Code mode”, with no Amp credits used. I ran amp --no-tui --runner-id ritza-mac in the project folder, and threads started from the Amp CLI or website ran on my Mac.

My first plan was to have Puck dispatch the work, since that’s what Puck is for, but on the free plan it refused.

So Amp’s dispatcher was an ordinary Amp thread instead. It created a git worktree per issue, added each one to the runner with amp runner dirs add, and started five threads with amp -m claude-code --executor runner:ritza-mac -x "<prompt>". Each thread appeared in the Amp web app as it started, which made it easy to follow along.

Dark mode was last because by the time it was ready, the filter and Lion’s Head changes were already on main, and its pull request conflicted. The agent rebased, resolved the conflict by switching the new filter buttons to the dark mode colours, and added the theme toggle to the new Lion's Head page as well. The dispatcher then checked that the live files matched main, removed the worktrees and runner folders it had created, and noted that it hadn't looked at the site in a browser.

Wave 2: nothing happened in the ten minutes after I opened the two new issues. Once I messaged the dispatcher, both were merged within two minutes and live within four and a half.

Amp is the only one of the three that can react to new work on its own. Orbs can wake on a schedule, on outside events, and through webhooks a plugin registers, and Amp works Linear issues natively. But the webhook docs describe them for Amp-managed orbs, which on Hobby are pay-as-you-go (from $0.08 an hour). On a free plan with your own machine, Amp waits for you like the others.

Herdr: multiple Claude Code agents in one terminal

Herdr runs in your terminal, so I opened it in a Terminal window in the project folder. On first launch, it offered to install “agent integrations”. For Claude Code, that adds four hooks to your global ~/.claude/settings.json, so Claude Code reports its status to Herdr directly instead of Herdr reading it off the screen. I installed it, and removed it afterwards with herdr integration uninstall claude.

The setup agent created the repo and confirmed the live site in 51 seconds. The dispatcher then ran herdr --skill, read the instructions, and used Herdr's own commands rather than plain git:

herdr worktree create --cwd "$PWD" --branch issue-1-lions-head-trail --base origin/main --label issue-1 --no-focus
herdr agent start issue1 --kind claude --pane w2:p1 -- --model claude-opus-5-5 --dangerously-skip-permissions
herdr agent prompt issue1 "Resolve GitHub issue #1 in hashkader47/factory-herdr. Follow AGENTS.md."
herdr agent wait issue1

herdr worktree create makes the git worktree and a Herdr workspace for it in one step, so every agent got its own named workspace in the sidebar, with a live working, blocked or done status. Of the three tools, this was the clearest view of what every agent was doing at any moment.

The dispatcher started all five agents within 30 seconds of being prompted, then waited on herdr agent wait in the background until they were done. No agent got stuck. One Pages deploy was cancelled because a newer merge replaced it, and the dispatcher spotted that and confirmed the later deploy had shipped the change.

Wave 2: again, nothing in ten minutes. After the prompt, both issues were closed in under two minutes, but only one by a pull request. The agent working on “Homepage feels a bit plain” in wave 1 had fixed the footer typo in passing, so the wave 2 agent checked main, found the fix, and closed the issue with a comment pointing at the commit. That's tidy, but it's also an agent going beyond "the smallest change that resolves the issue", which is worth knowing if your factory has stricter rules.

Herdr left the seven workspaces and worktrees open afterwards and offered to close them.

T3 Code: a GUI with a CLI behind it

T3 Code picked up my existing Claude Code login straight away. Its web interface on 127.0.0.1:3773 asks for a pairing token, so I used its command-line tool instead, which talks to the running desktop app. Setup took 59 seconds.

The dispatcher started each worker with one command, which created the worktree and branch for it:

~/.t3/bin/t3 thread start . "Resolve GitHub issue #1 in hashkader47/factory-t3. Follow AGENTS.md." \
  --provider claudeAgent --model claude-opus-5-5 --worktree main --branch issue-1-agent

Each thread appeared in the T3 Code app with its worktree, status and pull request number.

T3 Code was the fastest to a pull request every time, and the fastest to finish wave 1, but it also had the most confusing moment. The workers opened their pull requests and then ended their turns. One explained that it couldn’t sit and wait for CI in this environment, so it had asked T3 to wake it when the checks finished. The dispatcher saw four threads marked “completed” with open pull requests, decided they had stalled, and sent each one a message with t3 thread send telling it to merge.

Most of those messages weren’t needed: two workers had already merged by the time the message arrived, and replied to say so. The other two had conflicts by then and rebased. The dark mode agent, finding the filter buttons had arrived first, opened a second, follow-up pull request to give them dark colours, where Amp’s agent had fixed the same thing inside one pull request.

Wave 2: T3 Code started no threads in the ten minutes after the new issues were opened. Then GitHub suspended my account, so I couldn’t finish it.

Herdr vs T3 Code vs Amp: results side by side

The vague issue, “Homepage feels a bit plain”, got three different answers. Amp and T3 Code both added a gradient header banner. Herdr’s agent coloured the title and added a rule under the header instead. All three gave the cards a coloured edge by difficulty, which none of the issues asked for. These screenshots are each repo’s final main branch, served locally after the suspension took the live sites down.

Every agent in every tool used the same model and subscription, and one run per tool isn’t enough to rank them on speed. A dark mode rebase in one run and not another is the kind of thing that decides these times. What is consistent is that all three tools could be driven entirely by an agent, from the command line, on a free plan.

How my GitHub account got suspended

At 13:31 UTC, about an hour after the first repository was created, GitHub suspended my account and the three Pages sites went offline. GitHub hasn’t given a reason yet and I’ve asked support to reinstate it, but the hour’s activity looks like a bot: three new public repos, 21 issues, 19 agent-merged pull requests and about 40 Actions runs, all from one personal account. Every agent stayed inside its own test repository and stopped cleanly when requests failed.

If you build a factory like this, use a dedicated GitHub account or organisation, not your main one, and ramp up slowly. A suspension also cuts you off from every organisation you belong to.

Can you manage Amp with T3 Code or Herdr?

Herdr: yes. Amp is one of Herdr’s built-in agent kinds (herdr agent start --kind amp), and Herdr ships rules for reading Amp's working, blocked and idle states.

T3 Code: sort of. T3 Code doesn’t support Amp directly, but it can add agents from the Agent Client Protocol registry, and Amp is listed there as an “ACP wrapper for Amp”. It’s a community adapter built on Amp’s SDK, not something Amp or T3 publishes, and users in T3 Code’s issue tracker report that it streams whole blocks of output rather than token by token. I didn’t install it, because I couldn’t confirm it would use my Claude subscription rather than Amp credits.

Amp’s Puck: only Amp, and only paid. Puck manages Amp threads, not other agents, and on the free plan it doesn’t run at all.

Which one should you use?

• You live in the terminal and want to see every agent’s state at a glance: Herdr. It was the clearest view of five agents at once, its command-line interface is the most complete of the three, and it supports Amp out of the box.
• You want a polished app over the agents you already pay for: T3 Code. Worktrees and pull requests are built into every thread, it works on your phone, and its t3 command means you can script it too. It's early software, and it showed: agents ending turns while waiting for CI confused its own dispatcher.
• You want agents to keep working after you close the laptop, and you’ll pay for it: Amp. It’s the only one with cloud machines, schedules, webhooks and a manager agent. On the free plan you get none of those, but a good agent that runs on your own Mac with your own subscription, and a web and phone view of every thread.
• You want issues to become deployed code with nobody watching: none of them, for free. You’d need Amp’s paid orbs and webhooks, or your own trigger, such as a GitHub Action or a scheduled script that starts the dispatcher.

For more on the agents themselves, see our coding agent harness comparison, Claude Code vs Amp Code and Replit vs Amp.

Frequently asked questions

What about Paseo?

Paseo is closest to T3 Code and Herdr: a local service that runs Claude Code, Codex, OpenCode and others in git worktrees, with desktop, mobile, web and command-line clients, and built-in skills for agents to hand work to each other. I left it out to keep the test small. Its README doesn’t list Amp.

Is T3 Code free?

Yes. T3 Code is free and open source under the MIT licence, and it uses the subscriptions you already have for Claude Code, Codex and the other agents it supports. You can download it from t3.codes or the GitHub releases page, or install it on a Mac with brew install --cask t3-code.

Is Amp Code free?

Amp’s Hobby plan is free, and you can link a Claude or ChatGPT subscription so that Amp uses it instead of Amp credits. Running threads on your own machine as a runner is free. Puck, Amp’s own agent modes and Amp-hosted orbs need credits or a paid plan.

Is T3 Code any good?

For running several agents you already pay for, yes. In this test it was the quickest to get every issue to a pull request, and each thread showed its worktree, status and pull request in one place. It’s also early (version 0.1.5): its workers ended their turns while waiting for CI, which confused the dispatcher watching them, and its web interface needs a pairing token before you can use it in a browser.

How do you run multiple Claude Code agents with git worktrees?

Give each agent its own git worktree and branch, so they never edit the same checkout. Herdr does this with herdr worktree create, which also opens a workspace for the agent. T3 Code does it with --worktree and --branch on t3 thread start. With Amp, the dispatcher ran git worktree add itself and served each worktree to the runner with amp runner dirs add. In all three, agents that touched the same files still had to rebase when the first one merged.

Is Herdr better than tmux for Claude Code?

If you run several Claude Code sessions at once, yes. Herdr keeps tmux’s panes, prefix keys and persistent sessions, and adds agent status (working, blocked, done), notifications when an agent needs you, and an API that lets one agent start and prompt others. For a single session, tmux is enough.

Can Claude Code run multiple agents in parallel by itself?

Claude Code has its own subagents, but they run inside one session. To run several independent Claude Code sessions, each in its own worktree and each opening its own pull request, you need something like Herdr, T3 Code or a script around git worktrees. In this test, a single Claude Code dispatcher used each tool’s command line to start and watch the others.

Originally published at https://techstackups.com on November 2, 2026.

Herdr vs T3 Code vs Amp: Software Factory Test was originally published in Tech Stackups on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you're choosing an observability platform and you'll set it up with an AI coding agent, the agent's preferences affect what you end up with. We asked Claude Code which tools it recommends for metrics, logs and traces, and the same three names kept coming up: Grafana Cloud, Datadog and SigNoz. SigNoz placed first because agents recommended it over both rivals and it needed the least help from us to get running. Grafana Cloud placed second as the easiest to find, and Datadog came third.

Using Claude Code, we followed our audit method to assess their agent experience (AX) across four stages.

The four stages of an AX audit

The audit follows the funnel a developer goes through with their agent:

Explore the interactive audit questions for each stage.

• Awareness: Does the model already know about the product, without tools?
• Discoverability: Does it find the product when it searches online?
• Reputation: Does it recommend the product over alternatives, and get the facts right?
• Integration: Can it build and verify a small working integration?

Each stage gets a score from 1 to 4: GOOD (4) means little friction; OK (3) means minor gaps; POOR (2) means a partial result with significant gaps; FAIL (1) means the product is unknown or the task doesn't work.

How we tested

All testing used Claude Opus 5.5 (claude-opus-5-5) in Claude Code 2.1.308, on 28 October 2026. Every session started in a fresh folder with no audit context or user customizations. Each integration used its own fresh Claude Code profile with no MCP servers and only built-in skills and plugins, and we approved every command. Awareness questions ran with no tools, and discoverability and reputation questions with web search and web fetch only. We ran each prompt once, so treat these results as one developer's experience rather than a benchmark.

We chose the three by asking six questions about monitoring a production app and about Datadog's alternatives, then counting the products the answers named. Datadog and Grafana appeared in all six. Twice the agent's own shortlist was Grafana Cloud, Datadog and SigNoz, so we took SigNoz over New Relic and OpenObserve, which it listed only as other options when we described the job. The shared questions describe general observability needs (metrics, logs, traces and alerts, with OpenTelemetry and an MCP server for agents).

What are SigNoz, Grafana Cloud and Datadog?

All three collect your application's metrics, logs and traces, show them on dashboards and alert you when something goes wrong.

SigNoz is an open-source observability platform built on the OpenTelemetry standard, which stores all three kinds of data in one ClickHouse database. You can run it yourself for free with Docker, or pay for SigNoz Cloud from $49 a month with no per-user or per-host fees.

Grafana Cloud is Grafana Labs' hosted version of its open-source tools: Grafana dashboards plus separate stores for metrics, logs and traces, with a free tier that doesn't expire. Datadog is a hosted-only platform that covers monitoring, security and other products, each priced separately (mostly per host or per GB), with a 14-day free trial. Asked for Datadog's main competitors, our agent also named New Relic, Dynatrace, Splunk and Elastic. For a hands-on setup comparison of the same three on Kubernetes, see our beginner's guide to Kubernetes monitoring.

Results at a glance

The averages are not rounded. SigNoz and Grafana Cloud scored the same in two stages, and SigNoz moves ahead on reputation and integration.

SigNoz vs Grafana vs Datadog: which should you choose?

The agents' answer depended on what you care about most. Asked to pick between all three, the agent chose Grafana Cloud, and suggested SigNoz "if you want the simplest stack and the lowest bill, or you need to self-host for data residency" and Datadog "if budget doesn't matter much and you want the most polished product with the least work."

SigNoz was the agents' choice when OpenTelemetry is your standard and you want predictable costs. It won all four head-to-heads, and with no product named, the agent's pick was SigNoz "using its official MCP server." If you describe your problem broadly, expect SigNoz to come up only once you add specific requirements: it first appeared at the third question in two of our three search funnels. Our agent installed it locally with Docker and needed us only to register the first admin and copy one API key.

For Grafana Cloud, the agent's condition was "if Kubernetes depth, mature alerting and on-call, and a more established MCP server matter more." It appeared at the first question of every search funnel and won both head-to-heads with Datadog. Getting a working token took five rounds of help from us, partly because the Cloud Portal and each stack's own Grafana app both have pages for access policies.

Datadog came across as the most polished and most complete option, and the most expensive. Each head-to-head against Datadog came back to cost, and so did its risks answer: "The biggest risk with Datadog right now is cost. The product itself is mature and the company is stable." Our agent's script ran with the API key Datadog creates at signup, but that key can only send data, and reading data back needs a second key.

Awareness: does the model know SigNoz?

Awareness tests what a model knows from training data alone, with no web search.

We first asked for monitoring tools without naming any product. For a production web app's metrics, logs and traces, the model listed Datadog, Grafana Cloud, Honeycomb, New Relic and Sentry as hosted options, and SigNoz among the self-hosted ones. Its default picks were Grafana Cloud if budget matters, Datadog if cost is secondary, and SigNoz "If you want to self-host one simple system". When we added an MCP server for coding agents to the requirements, its answer was "My pick: Grafana Cloud", with Datadog as the alternative and no mention of SigNoz.

Then we named each product with its domain: "What is SigNoz (signoz.io)?", and the same question for Grafana Cloud (grafana.com) and Datadog (datadoghq.com). The model described all three accurately:

It also got SigNoz's background right ("Founded in 2021, backed by Y Combinator"). For Datadog, it cited "800+ integrations". Datadog's pricing page now says 1,000+.

Next we used each company's tagline from its own site, with no domain. The question for SigNoz was: "What do you know about SigNoz, the 'Observability for your team and AI agents'? What does it do, and who is it for? Say clearly if you don't know." Grafana Cloud's and Datadog's versions swapped in their own name and tagline:

For SigNoz, it guessed the tagline probably covered "an MCP server so coding agents like Claude Code can query traces, logs and metrics while debugging", which SigNoz does offer. For Datadog, it linked "AI-powered" to Watchdog and Bits AI.

Asked how to send a first OpenTelemetry trace from a Node.js app, the model gave step-by-step instructions for all three, which we didn't run. For SigNoz, it offered SigNoz Cloud or a self-hosted git clone and docker compose up route, which SigNoz has since deprecated in favour of an installer called Foundry.

What this means if you're choosing

Without web search, an agent knows all three and describes them correctly. It named Grafana Cloud and Datadog in both of our general questions, and SigNoz only as a self-hosting option.

Discoverability: can an agent find SigNoz?

Discoverability tests whether an agent with web search finds a product while looking for a tool like it. We repeated the two open awareness questions with web search, then ran three six-question funnels that narrowed a broad problem. The first four questions were the same for all three companies. Each funnel branched at question 5, where we asked for products using that company's tagline, and we named the company at question 6.

With web search, the open question about monitoring a web app produced a shortlist of all three: "If you want metrics, logs, traces and alerting in one place, I'd shortlist these three", with Grafana Cloud as "my default pick for most teams", then Datadog, then SigNoz. With the MCP requirement added, it answered "I'd use Grafana Cloud" and called Datadog "the strong second choice". SigNoz wasn't in that answer.

In funnel A, the agent recommended Grafana Cloud at question 2 and said Datadog was a poor fit for a team that doesn't want a huge monitoring bill. At question 3 it wrote: "Two products fit all your requirements well: Grafana Cloud and SigNoz Cloud. I'd pick Grafana Cloud". In funnel C, about monitoring costs, SigNoz was the pick: "For your size and stack I'd start with SigNoz Cloud and test Grafana Cloud against it."

Funnel B followed a developer who wants a coding agent to query production data, the audience SigNoz's tagline speaks to. Here it is on the SigNoz branch, with the start of each answer:

At question 3 in funnel B, Grafana Cloud was one of the six products the agent said fit. In its recommendations, though, it ranked Grafana Cloud last, after Honeycomb, Datadog, SigNoz Cloud and Dash0, and only with conditions, because of how its MCP server signs agents in:

Searching for Grafana Cloud's tagline, "open observability cloud", found Grafana Cloud in all three funnels. Datadog's "AI-Powered Observability and Security" found Datadog in one funnel, and in the other two the agent said the "closest match" was Dynatrace. SigNoz's tagline found nothing in any funnel, and each time the agent confirmed SigNoz uses that headline only after we named it. In one branch it said: "I should have checked SigNoz's homepage directly, since it was already on the list."

What this means if you're choosing

If you ask an agent to search for a monitoring platform, expect Grafana Cloud and Datadog first. SigNoz appears without being named once you add OpenTelemetry and specific requirements, and it was the agent's pick when cost was the main concern.

Reputation: do agents recommend SigNoz over Grafana Cloud and Datadog?

Reputation tests what an agent with web search says about each product once it's in the conversation. Every question described the same use case: "monitoring our Kubernetes services (metrics, logs, traces, alerts) with OpenTelemetry and an MCP server for agents".

Asked for the single best tool, with no product named, the agent read SigNoz's MCP server docs and chose SigNoz:

With all three named, it chose Grafana Cloud. It called SigNoz "a strong choice for a small team that is all-in on OTel", and said of Datadog: "It has the most mature MCP server (generally available since March 2026, remote, no setup)".

Grafana vs Datadog vs SigNoz: head-to-head results

Grafana Cloud beat Datadog in both orders, and SigNoz beat both of them in both orders. We asked each pair in both orders because agents can favour the product named first, and a pair counts as a win only if the same product wins both ways.

Five of the six answers opened by saying neither product was better for everyone. Against Grafana Cloud, the agent preferred SigNoz for one OpenTelemetry data store and simpler pricing, and Grafana Cloud for Kubernetes dashboards and alerting:

Against Datadog, both SigNoz and Grafana Cloud won on cost and on how Datadog handles OpenTelemetry data. The agent said "OTel metrics can also be billed as Datadog custom metrics, which is where many bills grow unexpectedly." Datadog's docs don't say this outright, but they class any metric that doesn't come from one of Datadog's own integrations as a custom metric.

The last question used each product's bare name: "What are the main risks or downsides of adopting SigNoz right now?", and the same for Grafana Cloud and Datadog. None of the names caused confusion.

For SigNoz, it also noted that ThoughtWorks put it in the "Trial" ring of its Technology Radar in April 2026, which we confirmed.

What this means if you're choosing

For an OpenTelemetry setup on Kubernetes with agent access, agents lean towards SigNoz, then Grafana Cloud. The agents recommended Datadog only when budget matters less than polish. Our use case favours tools built around OpenTelemetry, so a team already invested in Datadog's agent and integrations may hear a different answer.

Integration: can an agent build with SigNoz?

Integration follows three steps: sign up or install, get a credential for a script, then write and run a script that returns real data. Each company had one continuous session. The agents had shell access, web search and fetch, but no browser tool. We handled browser steps when asked.

The first request was: “Sign me up for SigNoz, or install it if that’s how it works, and get it ready to use. Do what you can yourself and tell me where you need me.” Grafana Cloud and Datadog received the same prompts with their names substituted.

None of the agents could create a key itself, so all three needed us for that step. We scored SigNoz OK because it needed no vendor account, and our part was a one-minute local registration and one copied key. Grafana Cloud and Datadog needed a browser signup first, and Grafana Cloud then needed five rounds of help with its token.

None of the three agents used the vendor’s own agent tooling, even though all three offer a Claude Code plugin and an MCP server, and Datadog and Grafana Cloud each have a command-line tool with a browser login (pup and gcx).

SigNoz: the agent installed it, then asked for one key

SigNoz’s agent chose to self-host. Its first attempt, a git clone of SigNoz's Docker setup, found the files had moved: "The docker setup is no longer in the main repo". It read the repo's deploy notes, then SigNoz's install docs, downloaded the new Foundry installer, read the script before running it, and started SigNoz in Docker. It was the only agent in this audit to read the vendor's docs during integration.

Its test trace failed until a first user existed, and the agent found the reason in the server logs before handing over:

For the key, it read SigNoz’s llms.txt and service-accounts page, then said: "I can't create the key myself, because only a signed-in admin can do that and I don't have your login." We created a service account and key in the local UI, as it described, and copied the key. The agent saved it to .env without printing it, re-sent its test trace and wrote a short Python script. Its output:

HTTP 200 — 1 service(s) in the last hour

SERVICE                   CALLS  ERRORS   AVG ms   P99 ms
setup-smoke-test              1       0     50.0     50.0

Grafana Cloud: signup, then five rounds for a working token

Grafana Cloud’s agent opened the signup page, tried to install Grafana Alloy (Grafana’s collector) with Homebrew, and used Docker when Homebrew failed on our outdated Xcode tools. For the key, it said:

Getting the token working took five rounds. We created an access policy and token from inside our stack’s Grafana app. The agent needed the stack’s metrics URL and user ID, and its directions pointed to the Prometheus card in the Cloud Portal, which we didn’t find because we were in the stack’s own app. The agent then tried to look the stack up with the token: first an empty list, then a 403. The stack-list address it called isn’t the one Grafana documents. It said: “I can’t tell from here why it’s refused, and I’d only be guessing at the cause.”

We copied the values from the stack’s Prometheus data source page instead. The URL there was an internal address, and the agent caught it: “the URL you pasted was Grafana’s internal address, which only their own servers can reach. I used the public address for the same server instead”. Its script then queried four metrics from Alloy, which it had already started sending to Grafana Cloud:

Targets up  [up]
  {'instance': '76193f0ba51f', 'job': 'integrations/self'} -> 1.00

Alloy version  [alloy_build_info]
  {'instance': '76193f0ba51f', 'job': 'integrations/self', 'version': 'v1.21.2'} -> 1.00

Datadog: signup, the default key, and a write-only script

Datadog’s agent checked the machine and handed over the signup: “Creating the account means choosing a password, confirming your email and agreeing to their terms, and I shouldn’t do those for you.” Asked for a key, it said “I can’t get into your browser session, so I can’t fetch the key from Datadog myself”, and asked us to copy one. Datadog had already created an API key when we signed up, so we copied that.

The agent read the key from the clipboard, tested it against each Datadog region to find ours, and saved it. Its script validated the key, sent a test metric and posted an event, which Datadog returned. Then it explained the limit:

Datadog’s docs confirm that reading data requires an application key as well as an API key.

What this means if you’re choosing

• SigNoz self-hosted needs no vendor account. Expect your agent to install it with Docker, then ask you to register the first user and copy an API key from the local UI.
• With Grafana Cloud, create the access policy in the Cloud Portal at grafana.com, where the agent expects it, and give your agent the stack’s URL and metrics user ID up front.
• With Datadog, copy the API key it creates at signup, and an application key too if your script needs to read data.
• None of the agents reached for the vendors’ plugins, MCP servers or CLIs on their own, so ask for them if you want them used.

Originally published at https://techstackups.com on October 28, 2026.

SigNoz vs Grafana vs Datadog for AI Agents was originally published in Tech Stackups on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you’re choosing between Voxtral TTS and ElevenLabs, it comes down to whose voice you need. With stock voices, ElevenLabs won every test I ran. It sounded human, followed every emotion tag, and got the name, date and order number right where Voxtral didn’t. Voxtral sounded rushed and flat, like text-to-speech from a couple of years ago. Cloning my own voice went the other way: Voxtral kept my South African accent, even from five seconds of audio, while ElevenLabs gave me a thick British accent.

Mistral released Voxtral 3 on 20 October 2026, and the TTS model is in public preview. I tested the things Mistral says are new against Eleven v4, using the same scripts I used in my Fish Audio and Gemini comparisons, so you can compare ElevenLabs across all three, and every clip is below to listen to.

Voxtral TTS vs ElevenLabs: the quick verdict

How we tested

• Models. Voxtral TTS 3 (voxtral-mini-tts-3-0) against Eleven v4 (eleven_v4).
• Through the APIs. Mistral AI Studio has a text-to-speech page, but its voice picker only offered preset voices, with no way to clone. So I generated everything through the two APIs: a Mistral API account with prepaid credits, and an existing paid ElevenLabs Creator account.
• Stock voices. ElevenLabs’ Global Artist — Newsreader against Voxtral’s Mary. Both are British female voices. I picked Mary from three Voxtral British voices as the closest match to the Newsreader.
• Same words on both. Only the tag syntax differs, because each tool writes tags its own way.
• Single take, default settings. Every clip is the first output from one generation. I allowed one deliberate fix per test, labelled separately.
• One listener. These are my impressions, apart from a transcription pass I used to check that no clip skipped any text.

Voxtral TTS vs ElevenLabs: natural delivery

What this tests: Mistral’s headline claim that Voxtral 3 voices “adjust their emotion, pace, and emphasis based on conversational context” with no instructions. The script is someone getting good news at work, then remembering it isn’t public yet, read as plain text with no tags.

I wasn’t going to tell anyone. We got the Lisbon project! I read the email three times before I believed it. Anyway, don’t tell anyone yet. It’s not official until Friday. Right. Now I just have to work out how to tell the team without grinning like an idiot.

Natural read

Plain text, no tags. Does the voice infer the excitement, the secrecy and the pause?

• Listen to Voxtral: Natural read (MP3) — voxtral-mini-tts-3-0. Rushed; excitement sounds forced
• Listen to ElevenLabs: Natural read (MP3) — eleven_v4. Conversational, shifts emotion on its own

ElevenLabs sounded more natural and more conversational. At 0:02 it puts secrecy into “I wasn’t going to tell anyone” and then switches to excitement, all inferred from the text.

Voxtral found the same moments: the excitement, the change at “Anyway” and the pause at “Right.” But it read the whole thing as if it were in a hurry, and the excitement on “Lisbon project” at 0:02 sounds forced. It sounded like someone reading from a sheet of paper. Its clip is 15 seconds against ElevenLabs’ 17, and you can hear the difference. I’d use the ElevenLabs take as it is.

Voxtral TTS emotion tags vs ElevenLabs audio tags

What this tests: Voxtral 3’s new inline tags. Mistral documents 34 emotions, pace control, and sounds like [laugh] and [sigh]. Voxtral has no whisper tag, so I swapped the whisper from my Fish Audio test for directions both tools support, and kept the same words and voices as before.

Voxtral tags last until the next tag of the same kind, so its script needs [pace=moderate] after "Friday" to get back to normal speed. ElevenLabs tags don't need resetting.

Directed read

Excitement, anxiety, a slow line, a sigh and a laugh, in the same places on both

• Listen to Voxtral: Directed read (MP3) — voxtral-mini-tts-3-0. Ignored 'anxious', overdid 'slow'
• Listen to ElevenLabs: Directed read (MP3) — eleven_v4. Every tag landed; the laugh carries over

ElevenLabs followed every direction and still inferred emotion where there was no tag, like the first sentence at 0:01. At 0:13 it moves from the sigh into the laugh naturally, and the laugh carries into the next sentence instead of stopping dead before it.

Voxtral’s emotions sounded read rather than acted. At 0:02 the [excited] sentence sounds the same as the untagged sentence before it. "Anyway, don't tell anyone yet" has no emotion at all, so [anxious] was ignored. Then at 0:09 it overdoes the slow line so much that I was waiting for it to finish, and against the rushed pace of everything else the change is very obvious. The sigh and the laugh both happen, but they sound like the model was told to do them. The pace reset did work: it went back to normal speed after "Friday".

Direction improved both tools compared with their natural reads, but Voxtral still felt rushed. ElevenLabs used the tags to colour the whole paragraph, while Voxtral applied each emotion to its own sentence and moved on. I didn’t hear either tool read a tag aloud.

Names, numbers and currency

What this tests: the kind of appointment message a clinic might send. This is where I expected Voxtral to do best, because Mistral says it benchmarked Voxtral 3 on numbers and unusual names, and on local formats like dates and currencies. Neither script had tags.

Hi, it’s Dr. Nkosi’s office. Your appointment is on 3 April 2026 at 14:45. The total is R1,249.50. Your booking reference is A-7419-X. If you need to change it, call 0800 555 0147. For the video appointment, please use the 2.4 GHz network if the other connection drops. Thanks, and we’ll see you on Friday.

Names and numbers

Name, date, time, rand amount, booking reference, phone number, frequency

• Listen to Voxtral: Names and numbers (MP3) — voxtral-mini-tts-3-0. Better amount and phone number, dropped a dash
• Listen to ElevenLabs: Names and numbers (MP3) — eleven_v4. Consistently correct

Both read 14:45 and 2.4 GHz correctly.

This was Voxtral’s best test, and I didn’t expect it to do this well after the first two. It read the rand amount and the phone number better than ElevenLabs. In my Fish comparison I wished ElevenLabs would say “oh-eight-hundred” for 0800, and Voxtral’s “zero eight hundred” is close to that. It was also less rushed here, at a perfectly reasonable speed.

But ElevenLabs was more consistently correct, and the error I care about most is Voxtral’s. Leaving out a dash in a booking reference is a problem when someone has to read it back. ElevenLabs’ take needs less editing before I’d use it.

Trying Voxtral’s pronunciation fix

Voxtral lets you tell it exactly how to say a word with phoneme_hints, which maps a word to IPA phonemes. Studio also has a Pronunciations panel. I gave "Nkosi's" the sounds ŋ k ɔ s i z and regenerated the same message once.

Listen to Voxtral pronunciation fix (MP3)

It didn’t work: the name changed from “nir-kow-see” to “nin-kaw-see”: the vowel is right now, but there’s an extra sound at the start. And because the fix regenerates the whole message, everything else changed too. The rand amount lost the “and” that I liked, and the booking reference lost both dashes this time. I couldn’t get the name right, and the takes were very inconsistent between generations, so for me it wasn’t worth the effort. You also have to write the hint in IPA, which most people won’t know.

Voxtral TTS voice cloning vs ElevenLabs voice cloning

What this tests: whether each tool sounds like me from a short recording, and keeps sounding like me when the mood changes. Mistral says Voxtral can clone a voice “from as little as 2–3 seconds of audio, capturing emotion, speaking style, and accent”. I’m South African, so accent drift is easy for me to hear.

I used the same recording of my voice as in my Fish and Gemini comparisons, but Voxtral rejected it:

Reference audio duration 33.90s exceeds the maximum allowed duration of 30.0s

Mistral’s docs say a reference can be “a few seconds to a few minutes”, so that limit came as a surprise. I cut the recording at a natural pause to 29.8 seconds and gave that file to both tools, so ElevenLabs had about four seconds less of my voice than in the earlier articles.

On Voxtral the recording is sent with each request and nothing is saved. On ElevenLabs I made a temporary instant clone with background noise removal off, then deleted it after the tests. Both clones read a passage I never recorded:

I left my keys next to the kettle again. If you’re heading out, could you check whether the back door is locked? I should be home before the rain starts. Oh, and the good news: they finally approved the project! I can stop refreshing my inbox every five minutes. Let’s get dinner somewhere nice tonight.

Instant voice cloning

Same 29.8 s recording of my voice, same new text

• Listen to Voxtral: Instant voice cloning (MP3) — voxtral-mini-tts-3-0. Keeps my accent and tone
• Listen to ElevenLabs: Instant voice cloning (MP3) — eleven_v4. Cleaner audio, British accent over mine

Voxtral wins clearly, because it kept my accent and tone. At 0:09 the word “finally” sounds very close to how I say it, and “They finally approved the project!” still sounded like me, without the rushed pace of its stock voice.

The ElevenLabs clone is the cleaner recording, but it doesn’t sound much like me. There’s a thick British accent laid over my own, and the expressive parts sound nothing like me. It’s the same thing I heard in my Gemini comparison. My impression is that ElevenLabs gives up a lot of the original voice to make the clone expressive, and the result sounds more like a stock voice. Voxtral struggles with expression, but it keeps what makes the voice mine.

If I had to use one of these clones, I’d use Voxtral.

Cloning from five seconds

To test Mistral’s short-reference claim, I also cloned on Voxtral from just the first 4.9 seconds of the same recording. ElevenLabs needs at least 10 seconds, so this one is Voxtral only.

Listen to Voxtral clone from five seconds (MP3)

It was surprisingly good, almost as good as the 29.8-second clone. It sounds a little flatter, but it still sounded more like me than ElevenLabs’ clone from six times as much audio. That’s one voice, mine, but it backs up Mistral’s claim.

Only clone a voice you own or have permission to use.

Voxtral TTS languages: a French test

Voxtral 3 supports 35 languages, with stock voices in 22 of them, so I generated a short French version of the Lisbon scene on both. It includes a date, a time and a euro amount written the French way. I don’t speak French well enough to judge accent or pronunciation, so there’s no verdict here, and you can listen and decide for yourself.

Je ne voulais le dire à personne. On a décroché le projet de Lisbonne! J’ai relu l’e-mail trois fois avant d’y croire. La réunion de lancement est le 3 avril à 14 h 45, et le budget est de 1 249,50 €. Bon, n’en parle à personne avant vendredi.

The stock voices are Voxtral’s Marie and ElevenLabs’ Claire, a native French library voice:

French, stock voices

Plain text with a date, a time and a euro amount

• Listen to Voxtral: French, stock voices (MP3) — voxtral-mini-tts-3-0
• Listen to ElevenLabs: French, stock voices (MP3) — eleven_v4

Mistral also says a cloned voice can speak languages other than the one it was recorded in. These are my two clones from the English recording, speaking French:

French, cloned voice

Clones made from an English recording, reading the same French text

• Listen to Voxtral: French, cloned voice (MP3) — voxtral-mini-tts-3-0
• Listen to ElevenLabs: French, cloned voice (MP3) — eleven_v4

The one thing I can judge is likeness: the Voxtral clone sounds like English-speaking me reading French from a piece of paper. I’m not sure that’s a good thing, but for a clone made from an English recording it’s more accurate than the ElevenLabs one.

Is Voxtral TTS free? Voxtral vs ElevenLabs pricing

Yes, you can create TTS clips with Voxtral Mini TTS 3.0 in Mistral Studio on a free account, with a 4,000-character input limit per generation. Free accounts don’t have API access. The UI doesn’t show a total free token or character allowance, so the 4,000-character limit shouldn’t be read as your total free quota.

I used an API account with prepaid credits for the comparisons in this article. In Studio, I could generate speech with preset voices but saw no cloning option. API prices below were checked on 26 October 2026, in US dollars; free-plan access was checked on 27 October 2026.

On paper, Voxtral is about six times cheaper than Eleven v4, but in practice I don’t think the gap is that big. Most of the ElevenLabs takes in this article were usable as they were, and none of the Voxtral stock-voice takes were, so you’d spend more generations getting a usable Voxtral clip. The cost per usable clip ends up closer than the list prices suggest.

For scale, these tests used 1,938 characters on my ElevenLabs account, about $0.16 at API rates. The Voxtral requests, including three short samples I used to pick a voice, came to about 2,600 characters, about $0.03 at list price.

Sources: Mistral API pricing, ElevenLabs API pricing, ElevenLabs plans.

A note for developers using Voxtral’s MP3 output

Voxtral returns MP3s without the header that tells a player how long the file is. My player showed the first clip as 16 seconds long, stopped at 14, and wouldn’t replay it. I fixed every Voxtral clip in this article by rewriting the header with ffmpeg -i in.mp3 -c:a copy -write_xing 1 out.mp3, which leaves the audio untouched. If you serve Voxtral MP3s directly to users, request WAV or fix the header first.

Is Voxtral TTS open source?

Voxtral TTS 3, the model I tested, isn’t: Mistral hasn’t published its weights, and self-hosting it means contacting Mistral’s sales team. The older Voxtral TTS from March 2026 has open weights on Hugging Face, but under a non-commercial licence (CC BY-NC 4.0), so you can run it locally for personal projects but not in a paid product. That March model is the Voxtral I included in my 16-provider blind TTS benchmark.

Is Voxtral a good ElevenLabs alternative?

For cloning your own voice, yes. For ordinary text to speech, not yet: ElevenLabs is significantly more refined and ready for commercial use, and Voxtral still feels slightly like a pet project. If you want something that works out of the box, use ElevenLabs.

• Voiceovers and anything with emotion: ElevenLabs. It sounded human, and it followed direction. Voxtral sounded flat and rushed in every stock-voice test.
• Prices, references and phone numbers: ElevenLabs. Voxtral read some numbers more naturally, but it dropped a dash in a booking reference and got the date order wrong.
• Cloning your own voice: Voxtral. It kept my accent where ElevenLabs didn’t, even from five seconds of audio.
• Cost at scale: Voxtral’s list price is far lower, but check how many takes you need before assuming it’s cheaper.

This covers two models, one listener and a handful of short tests. For a wider view, see my Fish Audio vs ElevenLabs and Gemini vs ElevenLabs comparisons, and for live conversation, the best realtime voice agent in 2026. I didn’t test Voxtral’s voice agent SDK, its professional voice cloning, or long-form narration.

Frequently asked questions

Is ElevenLabs good for voice cloning?

ElevenLabs makes clean, expressive clones, but they haven’t sounded like me. In this article, my Fish Audio comparison and my Gemini comparison, its instant clone of my South African voice came out with a British accent. Its professional voice cloning, which needs more audio, may do better, but I haven’t tested it.

Does Voxtral TTS support voice cloning?

Yes. Voxtral TTS 3 clones a voice from a short recording sent with each request, up to 30 seconds long, in MP3 or WAV. It worked through the API, on an account with prepaid credits, but the Studio website showed no cloning option. Mistral also offers professional voice cloning from 30 minutes to 3 hours of audio, which I didn’t test.

Is Voxtral better than ElevenLabs?

Only for cloning your own voice, where Voxtral’s clone sounded more like me. For ordinary text to speech with stock voices, ElevenLabs was better at emotion, direction and getting names and references right.

Originally published at https://techstackups.com on October 26, 2026.

Voxtral TTS vs ElevenLabs: Voice Cloning & TTS was originally published in Tech Stackups on Medium, where people are continuing the conversation by highlighting and responding to this story.

Python 3.15.1 came out on 21 October 2026. Two days later, I asked four search APIs built for AI agents for the latest stable Python release, and only one of them knew. Exa returned the 3.15.1 release page. Firecrawl, Tavily and DeepSeek returned pages that still named 3.15.0 or 3.14, and an agent reading those results would have given the wrong answer without knowing it.

Because an agent trusts whatever its search tool hands back, I ran 40 searches, 40 page fetches and two site crawls through all four tools. DeepSeek’s Search API is the newest of them, launched on 17 October at $0.75 per 1,000 calls, about a ninth of Exa’s price.

Exa returned the best search results for 31 of the 40 queries, so it’s the one to search with. For reading pages, use Firecrawl, which got usable content from 37 of 40 pages, including every PDF, and was the only tool that crawled a docs site reliably. DeepSeek was the cheapest and fastest at search, but it couldn’t fetch a single PDF. Tavily returned the leanest results, but it didn’t win any category outright. Everything ran on free plans, apart from five cents of DeepSeek credit.

What these tools actually do

All four give an AI agent access to the live web, through three kinds of call:

• Search: send a query and get back ranked results with page text, ready to drop into a model’s context. This is what Exa, Tavily and DeepSeek lead with.
• Fetch (or scrape): send a URL and get back the page as clean markdown, without the navigation, ads and scripts. This is Firecrawl’s core product, and the other three offer it too.
• Crawl: send a site and get back many pages. Only Firecrawl and Tavily do this.

A SERP API such as SerpApi is a different product: it scrapes Google and returns Google’s results as JSON. The four tools here run their own search index or a mix of sources, and return page content for a model to read. Our best SERP API comparison covers that category.

Firecrawl vs Tavily vs Exa vs DeepSeek: the quick verdict

How we tested

Plans. Firecrawl, Exa and Tavily on new free accounts, with no card. DeepSeek has no free tier for its API, so I topped up the $2 minimum, of which the tests used $0.05.

Search. 40 queries, 8 for each of five jobs, each run once, asking for 5 results. Firecrawl returned page content (scrapeOptions) for 32 of the 40 queries. Its free credits ran out before the 8 news queries, so those got titles and descriptions only. Exa used type: auto with page text, Tavily ran at both basic and advanced depth, and DeepSeek used need_content: true.

Grading. Each query’s five result sets were shuffled under anonymous labels and graded blind by an LLM, using a written rubric for each job. Every tool’s results were cut to the same 1,200-character excerpt: the opening plus the passages that best match the query’s words. Current-facts answers were checked against official sources on the day, such as python.org, the Federal Reserve and the South African Reserve Bank.

Fetch. 40 URLs, in four groups of 10: static pages, JavaScript-heavy pages, PDFs and bot-protected sites. For each page I loaded it in Chrome, captured three to five exact sentences from the main content, and checked whether each tool’s markdown contained them.

Crawl. docs.astral.sh/uv and docusaurus.io/docs, limited to 25 pages.

Single runs. Latency comes from one run per call, on 23 October 2026, so treat it as indicative.

Search results by job

Current facts: Exa found the answer nobody else did

What this tests: whether an agent could answer a factual question correctly from the results alone, when the answer changed recently.

Exa scored 8.4 out of 10 and had the right answer in all 8 result sets. For the Python question in the introduction, all five of its results were python.org pages, including the 3.15.1 release page.

For the other tools, the usual problem was stale pages mixed in with current ones. On the South African repo rate, every tool found the current 7.25%, but Tavily basic returned four pages still quoting the old 7%, which would mislead an agent that goes with what most of its sources say.

Breaking news: Exa first, DeepSeek a close second

What this tests: whether each tool surfaces coverage of an event from the last 48 hours, rather than older related stories.

I wrote the 8 news queries on the morning of the test from that week’s headlines, such as US jobless claims falling to 194,000 and Nvidia launching its DSX POD.

* Titles and descriptions only, because the free credits ran out.

For the jobless claims query, Exa returned AP, Reuters, CNBC and the Department of Labor’s own release, all with the 194,000 figure. DeepSeek also had the figure in every result, though two of its five were Bulgarian and Korean news sites. Tavily basic missed one query entirely, returning pages about AI in India for the Nvidia launch.

Docs lookups for coding agents: Exa, then Tavily

What this tests: whether a coding agent gets the official documentation page, with enough content to write correct code.

Firecrawl’s ranking was as good as Exa’s: the official page came first for 7 of 8 queries. Its score suffers because it returns whole pages, and on pages like Stripe’s webhook docs the start of the markdown was bot-check and navigation text. Tavily ranked official pages lower, but its short excerpts were cut around the part of each page that matched the query. One query asked about uuidv7, a PostgreSQL function that generates time-ordered IDs. Both Tavily modes returned the line from the official docs that shows how to call it, uuidv7([shift interval]), which is what a coding agent needs to use the function correctly. Exa ranked the same page first, but its excerpt left that line out.

DeepSeek was weakest on docs. For “verify Stripe webhook signature in Node.js”, it returned only third-party npm packages and no Stripe docs, and for the uuidv7 query it returned mailing-list threads instead of the documentation page.

Research and “find similar”: Exa won all 8

What this tests: open-ended discovery, where a good answer is a list of actual companies, papers or essays rather than articles about them.

Research showed the widest gap in the test: Exa won all 8 queries and returned 33 distinct matching entities. DeepSeek was next with 23, and Firecrawl and both Tavily modes were between 11 and 14.

For “open-source alternatives to Firecrawl for web scraping”, Exa returned five GitHub repositories. Both Tavily modes and Firecrawl returned vendor listicles titled “Firecrawl alternatives”. For “climate tech startups in Africa that raised funding in 2026”, Exa named five startups with 2026 rounds. DeepSeek returned four copies of the same press release about a fund.

Regional and non-English queries: Exa, with DeepSeek on Chinese

What this tests: South African local queries in English, plus queries in Chinese, German, Spanish and Japanese.

DeepSeek won the Spanish query outright, and tied with Exa on the Chinese query about the latest AI regulation, with current official Chinese sources. Most other tools leaned on 2023 law-firm summaries. On the German query about a Deutsche Bahn strike, DeepSeek did worst, returning Swiss and Austrian pages that recycled 2024 strike stories. Exa was the only tool to make it clear that there was no current Deutsche Bahn strike.

Every tool found local sources for the South African queries, and Exa and DeepSeek tied on Eskom’s 2026 tariff increase.

Search speed and size

Payload size matters because an agent pays to read every token. Firecrawl’s search returns the full page for each result. Those pages contained the right answer for all 8 current-facts queries, but at around five times Exa’s volume, and one search that hit a PDF returned 2.4 million characters.

Fetching pages: Firecrawl returned the most complete markdown

What this tests: give each tool a URL and check whether the returned markdown contains three to five exact sentences captured from the page in Chrome.

Firecrawl got content from Zillow, G2, Glassdoor and Takealot, which Tavily failed outright. It refused two sites by policy: Instagram returned “we do not support this site”, and a LinkedIn company page needs a separate enrichment setting turned on. The South African government site returned a server error to Firecrawl twice, while it loaded normally in a browser.

Exa was the fastest by a wide margin, probably because it can serve pages from its own index rather than loading them live. Its index also caught it out once: for the 100-page GPT-4 technical report, it returned a citation record with the author list and references instead of the paper.

DeepSeek returned a 404 for every PDF, which its docs list as the response for unsupported content types. On static pages it found 96% of the key facts, but its median fetch took 7.4 seconds.

Crawling a docs site: Firecrawl vs Tavily

What this tests: crawl a documentation site and return 25 pages. Exa and DeepSeek don’t offer crawling.

Tavily’s crawl followed links off the site by default, into GitHub and the documentation theme’s own site. Its allow_external: false option, which should keep it on the site, returned at most one page in every run. Tavily's map endpoint found the right 20 to 25 URLs on both sites, so it can discover the pages, and the failure is in the crawl step.

Each Firecrawl crawl cost 25 credits, and I only ran it once per site because the free credits had run out. Tavily’s empty runs weren’t billed.

Firecrawl, Tavily, Exa and DeepSeek pricing: is Firecrawl free?

All four are free to start except DeepSeek. Prices below were checked on 23 October 2026, in US dollars.

Firecrawl charges per PDF page

The free plan’s 1,000 credits went much faster than I expected. The 194-page RFC 9110 PDF cost 194 credits on its own, and the GPT-4 technical report cost 100. The 10 PDFs used 564 credits, more than half the month’s allowance. Firecrawl’s billing docs do list this, as an extra credit per PDF page, but it’s easy to miss when the headline is “1 credit per page”. By the end of the test, Firecrawl had used 992 of its 1,000 free credits.

The others used much less:

• Tavily: 194 of its 1,000 credits.
• Exa: $0.32 of $20.
• DeepSeek: $0.05 of my $2 top-up.

What 1,000 searches costs

DeepSeek Search API: what you need to know

DeepSeek’s Search and Fetch APIs launched on 17 October 2026. Each is a single POST to api.deepseek.com/v1/tools/search or /v1/tools/fetch with your API key.

• Search results carry more than the others’ do. Each result has a short snippet, a summary of around 600 to 800 characters, the full page if you ask for it, a publication date, an authority score from 0 to 4 and a rerank_score from 0 to 3.
• Price doesn’t change with result count. Asking for 50 results costs the same $0.75 per 1,000 as asking for 1. Failed and empty calls aren’t billed.
• Queries are capped at 128 characters. That’s fine for keyword search, but too short for a long natural-language question.
• It doesn’t de-duplicate URLs. For “Firecrawl Hobby plan price”, it returned Firecrawl’s pricing page three times, each with a different affiliate code in the URL. Across 40 queries it returned 7 duplicates, the most of the four.
• Your data is stored in China. DeepSeek’s privacy policy says it stores personal data in the People’s Republic of China, and that it shares search keywords with unnamed third-party search APIs. A Search Engine Journal report notes that a provider called Bocha names itself as DeepSeek’s web search supplier. Check this with your compliance team before sending customer queries.

Using DeepSeek search inside Claude Code

DeepSeek’s docs say its search is free until 1 November when called through Claude Code, Codex or DeepSeek’s own harness. That works by pointing Claude Code at DeepSeek’s API, so DeepSeek’s model replaces Claude in that session. I tried it first, and it isn’t the same as using the API:

• Claude Code’s WebSearch tool passed only titles and URLs to the model, without DeepSeek’s summaries, page content or scores.
• WebFetch fetched the page itself rather than calling DeepSeek’s Fetch API. Asked for a Paul Graham essay verbatim, it returned a summary instead, saying quotations were limited to 125 characters.
• The search calls were free, but the model’s tokens weren’t, and a new account has no starting balance. My first attempt failed with 402 Insufficient Balance until I topped up.

If you want DeepSeek’s results in an agent, call the API directly.

Using them with AI agents

All four publish an llms.txt file for coding agents to read. Firecrawl, Exa and Tavily each have an official MCP server that covers search and fetch. Firecrawl's and Tavily's servers also cover crawling. I couldn't find an MCP server from DeepSeek. For deeper agent setup testing of two of these tools, see Firecrawl vs Tavily.

Which should you use?

• Search for an AI agent: Exa. It won on current facts, news, research and non-English queries, and its results are compact enough to put straight into context.
• Reading specific URLs, PDFs or protected pages: Firecrawl. It returned usable content from the most pages and handled every PDF. Budget for PDFs, which cost a credit per page.
• Crawling a whole site: Firecrawl. Tavily’s crawl wandered off-site or came back empty in our runs.
• Lowest cost at high volume: DeepSeek, if your queries are public data and data stored in China is acceptable. It was fast and fresh on news, but weaker on docs lookups and it can’t fetch PDFs.
• Smallest, cheapest results to feed a model: Tavily. Its basic search returned about 5,600 characters and did well on docs lookups, though it didn’t win any category outright in our tests.

Frequently asked questions

How much does Firecrawl cost?

Firecrawl’s free plan gives you 1,000 credits a month without a card, and Hobby costs $19 a month for 5,000 credits. A search costs 2 credits per 10 results, a page costs 1 credit, and each PDF page costs another credit, so long PDFs use credits fast.

Is there a free web search API?

Three of the four have free plans with no card: Tavily gives 1,000 credits a month, Firecrawl 1,000 credits a month, and Exa $10 of credit a month. DeepSeek has no free tier for direct API calls.

What are the best alternatives to Firecrawl?

For search, Exa returned better results in our tests, and DeepSeek is much cheaper. For turning pages into markdown, none of the alternatives we tested matched Firecrawl: Tavily found 74% of the key facts, Exa 69% and DeepSeek 41%, against Firecrawl’s 85%.

Is Tavily search better than Exa search?

No. Exa produced the best results for 31 of our 40 queries, against 7 for Tavily basic and 5 for Tavily advanced, and it scored higher on every type of query. Tavily came closest on docs lookups, where it averaged 7.5 against Exa’s 7.8.

Originally published at https://techstackups.com on October 23, 2026.

Firecrawl vs Tavily vs Exa vs DeepSeek: Best Search API for AI Agents (2026) was originally published in Tech Stackups on Medium, where people are continuing the conversation by highlighting and responding to this story.

Google released Gemini 3.8 Flash TTS on 23 September 2026. ElevenLabs released Eleven v4 five days later. I put both through the same five tests, covering emotional tags, hard-to-say text, long-form narration, a two-person argument, and cloning my own voice.

ElevenLabs won three of the five (hard text, long-form, dialogue). Gemini won two: emotion, and voice cloning, where it kept my South African accent and ElevenLabs gave me a British one. If you came here for the ElevenLabs voice cloning question specifically, that result is the interesting one, and it’s further down.

Google ships a “Flash TTS” and a “Flash-Lite TTS” under the same 3.8 label, which is confusing. I tested Flash, and Eleven v4 on the ElevenLabs side.

Gemini vs ElevenLabs: the quick verdict

How we tested

This is one person listening. I tested in English only, and I ran each scenario once with default settings.

• Same text, same direction. Both models got identical text. Direction went inline, sentence by sentence ([whispering] for ElevenLabs, <whispering> for Gemini). I left Gemini's separate Style instructions box untouched so the comparison is like for like. (My first Gemini attempt used one overall style line and came out hushed throughout, so I threw it out as a test-design flaw.)
• Free-tier web UIs for everything except cloning: ElevenLabs with v4 selected, and Google AI Studio.
• One caveat in ElevenLabs’ favour. It returns two outputs per prompt and I picked the better one. Gemini returns one.
• Cloning ran through the APIs, because AI Studio wouldn’t run cloning without billing and instant voice cloning on ElevenLabs needs a paid plan in the UI. I cloned my own voice from the same 29.5 second sample on both. A friend listened to the cloning results too, so that verdict isn’t mine alone. The other four are mine only.
• Voices. Scenarios 1 to 3 used Gemini’s Cleo (warm and engaging) against ElevenLabs’ Lauren (friendly, comforting and soft). The dialogue test added Mako and Jack John.

Tested in early October 2026

Gemini voice cloning vs ElevenLabs voice cloning

What this tests: can the model sound like a specific person from about 30 seconds of audio, and keep that identity when the line changes from neutral to emotional to narration. I’m South African, which makes accent drift easy to hear.

Gemini asks for a 10 to 30 second sample plus a spoken consent recording, and the consent has to come from the same voice as the sample. ElevenLabs takes just the sample. Creating the clean clone took 5.54 seconds on Gemini and 3.23 on ElevenLabs.

Listen to Gemini and ElevenLabs voice clones on Techstackups: neutral, emotional and narration samples

Gemini wins this one clearly. Its clones kept my accent and tone far better. The ElevenLabs clones kept a slight resemblance to my voice, but it sounded like my voice and tone had been overlaid on someone else’s accent, a heavily British one. That held across all three lines, so it looks like a general problem with ElevenLabs cloning on my sample. Neither model captured expression well from a 30 second sample, though Gemini was less bad, and its emotional line didn’t convey the emotion accurately. The Gemini narration clip had the best voice and accent match of anything I tested, but it sounded damp and fuzzy next to Gemini’s other clips, like a recording of me.

What happens with a noisy sample

I also recorded the same text in a noisier setting. Gemini rejected it outright at clone creation:

Voice mismatch detected. The speaker in the consent audio does not match the speaker in the voice sample.

My consent clip was recorded in a quiet room, so Gemini was comparing a quiet recording against a noisy one and decided they weren’t the same person. That’s a useful safeguard and an annoying one if your only sample is noisy.

ElevenLabs accepted the same noisy sample with no consent step. The clone was much worse than its clean one: a very slight resemblance, and bad accent and expression. Here’s the noisy narration, ElevenLabs only, since Gemini produced nothing:

Hear the ElevenLabs voice clone made from a noisy recording on Techstackups

Only clone a voice you own or have permission to use. Gemini enforces consent at this step and ElevenLabs doesn’t, which matters if you’re building a product that lets users clone voices.

Expressive tags and emotion

What this tests: does the model perform the whisper, the laugh and the sigh, or just read the word, and how smooth is the shift between them. The script runs from a whisper to excitement to a laugh to a voice cracking with emotion.

Compare Gemini and ElevenLabs on whispers, excitement, laughter and emotion on Techstackups

Gemini sounded significantly more natural. It inferred pauses from context, and the pauses sound like someone talking to a person and waiting for their reaction. My favourite moment is at 0:16. The line is “We’re giving you the Lisbon project,” and Gemini went from a whisper to a slight exclamation with no emotion tag at all, purely because it understood the news was exciting. The worst moment was ElevenLabs at 0:18, where the shift from laugh to whispering is jarring.

My read on why: ElevenLabs seems to treat the emotion tag as a parameter and then read that sentence with it. Gemini seems to read the current sentence together with the next one, so the emotion spans both. That is my impression from listening. I didn’t hear any tag spoken aloud in this test.

Hear Gemini and ElevenLabs perform a two-person argument with an interruption on Techstackups

ElevenLabs won the dialogue test by a distance. On Gemini the interruption feels like a pause, and the emotion lands as a pre-sentence emote. For the [laughs, then quietly] tag, Gemini produced a normal laugh and then kept speaking loudly, ignoring "then quietly". ElevenLabs produced a laugh that fit the conversation. At 0:12 it also inferred the severity of the emotion from context, which it did well in every scenario. Neither is perfect, and I flagged robotic glitches and some overacting. Generation took 14.55 s on Gemini and 6.26 s on ElevenLabs.

Numbers, names and hard text​

What this tests: the stuff that breaks customer-facing voice. An order number, a date, a currency amount, a phone number, a URL, and a run of hard names. No tags on either side.

The rest were fine on both: $1,249.50, R22,800, acme-labs.io/returns, 14:45, “Thursday the 9th,” HTTP 404, 2.4 GHz, Xiaomi, Worcestershire, Leicester and Edinburgh. I’m not giving a score.

Compare how Gemini and ElevenLabs read numbers, names and difficult text on Techstackups

ElevenLabs wins, mostly on the dropped dash in the order number and the Irish name. But remember the two-takes caveat: ElevenLabs gave me two outputs and I chose the better one. Generation took 18.88 s on Gemini and 17.53 s on ElevenLabs.

Long-form narration

What this tests: about 2.5 minutes of Dickens (the opening of Great Expectations), plain text with no tags, to see whether pace creeps, the voice changes character, or the tone jumps.

ElevenLabs handled it without trouble. No pace creep, the voice kept its character, the paragraph breaks sounded intentional, and there were no sudden tone or loudness jumps.

Gemini failed on my first day of testing. It would speak only the first four to six words of the passage and then fail, repeatedly. I lost several retries to it.

I retried the next day and the full passage generated. I don’t know why it failed the first time. But even the successful run has a problem: between 2:06 and 2:08 the tone drops jarringly, in a way that doesn’t match the emotion the surrounding text calls for. Generation took 45.9 s on Gemini and 48.28 s on ElevenLabs.

Listen to the Gemini and ElevenLabs long-form narration comparison on Techstackups

ElevenLabs was the better read of context here. I was impressed at how well it inferred expression from the text alone.

Which is the best TTS model? Which is the best voice cloning model?

This covers two models. For the 16-provider view, see Best TTS in 2026: blind benchmark, and for the realtime and price angle see Cartesia vs ElevenLabs. Within these two:

• Creators and audiobooks: ElevenLabs. It stayed consistent across 2.5 minutes of narration and Gemini didn’t.
• Customer-facing voice and IVR: ElevenLabs, with caveats. Gemini’s phone-number read was better, but the dropped dash in an order number is the kind of error that costs real money.
• Developers watching API cost: Gemini, by a wide margin, if its long-form reliability holds up for you.
• Voice cloning: Gemini, at least for an accent like mine and a clean sample. It also enforces consent. ElevenLabs may do better with other voices, but I only cloned one.
• Dialogue and character voices: ElevenLabs.
• Expressive reads where you want the model to infer emotion: Gemini.

My recommendation

I’d use ElevenLabs for anything that has to be right first time: narration, dialogue, and text full of numbers. I’d use Gemini when cost matters, when I want a model that reads emotion across sentences, or when I need a clone of a real voice that keeps its accent. If you take one thing from this, take the cloning result. On a clean 30 second sample, Gemini kept me sounding like me while ElevenLabs didn’t. Test with your own voice before you commit.

I didn’t test voice design from a text description, re-roll consistency, other languages, or the UI friction of each tool.

Originally published at https://techstackups.com on October 2, 2026.

Gemini vs ElevenLabs Voice Cloning and TTS (2026): Gemini 3.8 Flash vs Eleven v4 | Tech Stackups was originally published in Tech Stackups on Medium, where people are continuing the conversation by highlighting and responding to this story.

Monospace is a new product for giving AI agents safe access to company data, from Monospace Inc., the company behind Directus. In our tests, a coding agent connected to it over MCP and answered questions about our data correctly. But the agent only found Monospace when we searched for its tagline, and for Monospace’s own use case it recommended other tools, including Directus.

Four days after launch, we put Monospace through our agent experience (AX) audit, following our AX audit method. The audit has four stages:

• Awareness: does the model know Monospace without searching?
• Discoverability: can an agent find Monospace when it’s looking for a tool like it?
• Reputation: does the agent speak well of Monospace, and recommend it over other options?
• Integration: can the agent build a working proof of concept with Monospace?

How we tested

All testing used Claude Opus 5.5 (claude-opus-5-5) in Claude Code 2.1.285, on 6 October 2026. Every session started in a fresh, empty folder with no project files, memory, MCP servers, skills or plugins. Awareness questions ran with no tools at all, and discoverability and reputation questions with web search and web fetch only. We ran each prompt once, so treat these results as one developer's experience rather than a benchmark.

Scores at a glance

Agents don’t know Monospace, and only find it when they search for its own tagline. Once they do find it, they like the design but hedge every recommendation, and often fill gaps with facts about Directus. In the integration, the agent installed Monospace, loaded our data and answered questions over MCP with a few short human steps, but the free plan blocks the governance features Monospace leads with.

See our AX audit rubric for how we score each stage.

Awareness: does the model know Monospace?

Awareness tests what a model knows from training data alone, with no web search. For a product launched four days earlier, we expected the answer to be no. What matters is how the model fails, whether it admits it doesn’t know, makes something up, or confuses Monospace with something else.

We started with two questions that describe Monospace’s pitch without naming it. Each answer below comes from a fresh session with no tools.

The first answer describes the gap Monospace says it fills, and the second picks a competitor with confidence. Then we named Monospace:

Given the domain, the model knew the company but not the product. Given the tagline, it said it didn’t know. It never invented features, which is good behaviour, but a developer asking their agent about Monospace today either hears about Directus or gets “I don’t know”. In other sessions the agent also raised two name clashes, monospace fonts and Google’s old internal “Monospace” codename for what became Project IDX.

What Monospace can do about it

Training data catches up slowly, so there’s no quick fix here. The useful work is making sure the next training cut sees Monospace as a product in its own right. That means launch posts, guides and comparisons that pair the name with what it does (“Monospace, the governed API layer from Directus”).

Discoverability: can an agent find Monospace?

Discoverability tests whether an agent with web search finds the product when it’s looking for something like it. We ran three six-question “funnels” in a single session each. Each funnel starts with a broad problem and narrows towards Monospace one question at a time, then names it at question 6. We recorded the first question where Monospace appeared.

Each funnel asks from a different angle, because different people describe the same problem differently.

Here are funnel A’s six questions, asked in one session, with the start of each answer.

At question 3, the agent searched the web for a product that does what Monospace does and concluded that none exists. In funnel C it said the same thing in different words: “No product does this as a drop-in layer, and a gateway can’t do it reliably.”

Funnel B, the developer’s angle, went worse. At question 3, the agent recommended Monospace’s parent company’s older product for Monospace’s job:

And searching for Monospace’s own tagline didn’t find it either:

Once found, Monospace became the agent’s top pick in all three funnels. It called Monospace “the closest match to your requirements of anything I’ve found” in funnel A, and “the best conceptual fit of everything we’ve looked at” in C. In B it said “the product is brand new but the team isn’t.” Each time it added the same caveat, that Monospace was days old and unproven.

What Monospace can do about it

Publish pages built around the problems agents are asked about.
Agents ran searches like MCP gateway agent identity fine-grained authorization audit log Postgres SaaS 2026 and AI agent data access control plane field-level permissions MCP new product 2026, and none of them surfaced Monospace. Guides and docs titled around those problems, in a developer's words, would give agents something to find.​

Get into the comparison articles agents already read.
The recommendations agents made instead came from vendor comparison posts and round-ups, Bytebase and Cerbos blog posts, Product Hunt, and MCP directories like Glama. That’s where Hasura, WunderGraph Cosmo, DreamFactory and Cerbos got their mentions.

Separate Monospace from Directus on purpose.
In funnel B the agent recommended Directus for Monospace’s use case. A clear “Monospace vs Directus: which should I use?” page on both sites would teach agents when to send developers to which product.

Reputation: do agents recommend Monospace?

Reputation tests what the agent thinks once Monospace is named, including whether it recommends it over competitors and whether it gets the facts right. We ran eight questions, each in a fresh session with web search. Hasura and WunderGraph Cosmo were among the competitors the agent named most often before it found Monospace. We added Directus on purpose, because of funnel B.

Whenever the agent found Monospace, it liked the design. For a regulated company, it called it “the right design for a regulated environment: agents never hold database credentials directly.” But it never recommended Monospace without conditions, and most verdicts came back to how new it is.

The answers’ quality depended on which pages the agent reached. The most accurate answer, the startup “Not yet”, came from Monospace’s own homepage, pricing page and connectors page. It found that Stripe and HubSpot are listed as “Coming Soon”, and that row/column-level security and audit logs aren’t on the free plan. When the agent read less, it filled the gaps with outdated or borrowed information:

The two questions that named Monospace without its domain went worst:

For the risks question, the agent searched for “Monospace developer tool” and “Monospace launch 2026”, and neither search found the product. The pricing comparison never reached monospace.io.

Monospace vs Directus

Directus is Monospace Inc.’s older product, and it has its own MCP server. When we asked whether to choose Monospace over Directus for exposing Postgres and internal APIs to apps and agents, it leaned towards Monospace, “mainly because you need internal APIs as well as Postgres”. Its reasoning was that Directus is built around a single SQL database, while Monospace applies one permissions model across several sources. It described the two as “a sibling product, not an unknown vendor”, and still pointed teams that only need Postgres towards Directus, for its proven field-level permissions and its admin UI.

What Monospace can do about it

Put the key facts on the pages agents read first.
The homepage was the page agents read most, and it doesn’t name Postgres or state a licence. Agents that stopped there guessed both. A line on the homepage naming the supported databases, the licence and where pricing lives would have fixed several wrong answers.

Settle “public preview” or “GA”.
Monospace’s launch article calls it a public preview, and the Directus community announcement calls it GA. Agents picked up both (“Their own posts disagree on whether it’s “GA” or “public preview””), and used “preview” as a reason not to recommend Monospace for production. One status everywhere would remove that reason.

Publish what agents told our test user to ask sales.
For the regulated company, the agent listed five questions to ask Monospace’s sales team. They covered whether audit logs record the full query, whether the log is tamper-evident and can stream to a SIEM, whether SAML or OIDC work self-hosted, what licence covers self-hosting, and what compliance reports exist. Docs pages that answer those would give agents what they need to move past “don’t approve it yet”.

Integration: can an agent build with Monospace?

Integration tests whether an agent can install the product and build a “hello world” with as little human help as possible. We gave it three spreadsheets for a fictional business (50 customers, 30 products, 300 orders with 741 line items) and asked it to:

1. Set up Monospace and give us a localhost link.
2. Get the spreadsheet data into Monospace and connect itself to it.
3. Answer a question using Monospace: which region had the most revenue in the last three months, and who were its top three customers?
4. Create a restricted key for an “analyst” agent that can’t see customer emails, and prove it.
5. Mark one order as refunded through Monospace.

We ran it in Claude Code’s default permission mode, approving each command, so we could stop the agent from reading anything outside its folder. Monospace has no hosted signup (“Get Started” leads to a contact form), so the agent had to self-host it with Docker.

Installing: one hint, then four minutes

We replied “It’s the one at monospace.io.”, and the agent then read the installation page, downloaded the official Docker Compose file, and had Monospace v1.0.0 running with Postgres, RabbitMQ and Redis in 4 minutes 12 seconds, with no errors.

Creating the admin account, accepting the licence agreement and creating a workspace took us 6 and a half minutes in the browser, with no email verification. The last onboarding screen shows how to connect Claude Code (a claude mcp add command) and how to install Monospace's agent skills plugin.

Loading the data and connecting over MCP

Monospace’s homepage illustrates its data sources as files like customers.xlsx and inventory.xlsx, but its connectors are PostgreSQL, MySQL and MariaDB, with no spreadsheet connector. The agent worked this out from the docs and took a sensible route. It converted the spreadsheets to SQL, loaded them into a new database inside Monospace's own Postgres container, and registered that database as a data source through Monospace's API. Monospace detected the relationships between the tables on its own, and the agent checked that every table's row count matched the spreadsheets.

The agent leaned heavily on Monospace’s agent-friendly docs, including llms.txt, the 1.4 MB llms-full.txt, and raw Markdown versions of each docs page. The agent said the OpenAPI spec served by the running instance "mattered most". The docs have no request examples for creating a data source, so it worked out the format from that spec, by trial and error.

It then needed one thing it couldn’t do itself, an API key. It asked us to create one in the UI and write it to a file with a ! pbpaste command, so the key never appeared in the chat. With a key in place, the agent tested Monospace's MCP server by hand with curl and found 7 tools. It then wrote a project-level .mcp.json that reads the key from the file at runtime, so the key isn't stored in the config either. Claude Code needed a restart to load the new MCP server.

The hello world: a correct answer

Every figure matched what we calculated from the spreadsheets beforehand. The agent stated its date window and which order statuses it counted, showed how the ranking changes if pending orders are included, and cross-checked the result in Postgres.

Monospace can’t calculate totals itself (its docs have a page called “No Aggregates”), so the agent pulled the rows through MCP and added them up in a Python script. That’s fine for 300 orders, but a bigger dataset would mean moving a lot of rows through the agent’s context.

Field-level permissions: blocked by the free plan

Asked for a restricted “analyst” key that can’t see customer emails, the agent read the access control docs, built a policy, and tried to create it. Monospace refused, and the agent queried Monospace’s licence API to find out why:

It confirmed nothing had been half-created, and declined to fake the result by hiding the column in Postgres instead, saying that “isn’t what you asked for”.

This matches Monospace’s pricing page, which lists row/column-level security, service accounts and custom roles and policies as Business-plan features. But it means the features Monospace leads with, per-agent identities and field-level rules, can’t be tried on the free self-hosted version without contacting sales. The agent also noted that the licence limits weren’t on any docs page it read. It only found them through the API, after being refused.

A live write: done

What it took

The whole session took about an hour and a half, including our steps. The agent never used Monospace’s agent skills plugin, SDK or CLI. It skipped the skills plugin because “it wasn’t needed for this”.

What Monospace can do about it

Let people try governance without a sales call.
A trial licence that a developer can get without contacting sales, or a small allowance of custom policies and service accounts on the free plan, would let agents demonstrate per-agent identities and field-level rules.

Document the free plan’s limits where agents look.
Put the Starter limits (0 custom policies, roles and service accounts) in the docs next to the access control and API key pages, so an agent can tell a user before it builds a policy that can’t be saved.

Add request examples for the API calls agents need.
Creating a data source and writing a policy were the two places the agent had to guess. One complete example request for each would remove most of the trial and error.

Make the API keys page point to personal keys.
On the free plan, the prominent API Keys page leads to a paywall, while the working option is under Account → Access. A link from one to the other would save the confusion.

What this means for Monospace

Monospace has built a lot for agents. It has an MCP server, llms.txt and llms-full.txt, Markdown docs, an OpenAPI spec on every instance, an agent skills plugin, and an onboarding screen that tells you how to connect Claude Code. Once an agent was connected, it loaded data, answered questions correctly and made live writes, all through MCP.

The problems come before and around that:

• Agents only found it by searching for its tagline, and in one test the agent recommended Directus for its job.
• Without the domain in the prompt, the agent either couldn’t identify it or priced Directus instead.
• Field-level permissions and agent identities can’t be tried on the free plan.

The first two are fixable with content, such as pages built around the problems agents get asked about, clear “Monospace from Directus” naming, and the key facts on the homepage. The third is a pricing decision, but documenting the limits would at least stop agents and developers from running into them by surprise.

Cloudflare’s isitagentready.com, which checks how readable a website is for AI agents, scored Monospace’s docs at docs.monospace.io 20 out of 100. Those are the docs the agent relied on in the integration, through their llms.txt and Markdown pages.

Originally published at https://techstackups.com on October 6, 2026.

Monospace from Directus Review: An AX Audit | Tech Stackups was originally published in Tech Stackups on Medium, where people are continuing the conversation by highlighting and responding to this story.

SpacetimeDB’s homepage is written for AI agents. Its headline is “Let AI ship the whole backend, glue-free”, and the site ships an llms.txt, an agent setup guide, a Claude Code plugin, agent skills, an MCP server built into its CLI, and a public benchmark of how well LLMs write code for it.

In our tests, a coding agent built a working, live-updating app on SpacetimeDB with no help from us. It used none of that tooling. And when we asked agents which backend to use for a web app, they picked Convex or Supabase.

SpacetimeDB is a database that is also your server, made by Clockwork Labs. We put it through our agent experience (AX) audit, following our AX audit method. The audit has four stages:

• Awareness: does the model know SpacetimeDB without searching?
• Discoverability: can an agent find SpacetimeDB when it’s looking for a tool like it?
• Reputation: does the agent speak well of SpacetimeDB, and recommend it over other options?
• Integration: can the agent build a working proof of concept with SpacetimeDB?

How we tested

All testing used Claude Opus 5.5 (claude-opus-5-5) in Claude Code 2.1.285, on 7 October 2026. Every session started in a fresh, empty folder with no project files, memory, MCP servers, skills or plugins. Awareness questions ran with no tools at all, and discoverability and reputation questions with web search and web fetch only. The integration ran in a fresh Claude Code profile with no settings, plugins or history, with every command approved by us. We ran each prompt once, so treat these results as one developer's experience rather than a benchmark.

Scores at a glance

The model knows SpacetimeDB well, but as the game backend it was in 2025. It doesn’t know the 2026 rebrand, the push into web apps or the agent tooling. Agents find it straight away for multiplayer games, but for web apps it comes second to Convex, and agents recommend Convex or Supabase over it. In the integration, the agent installed it, built a live page and hid private data without any help from us.

See our AX audit rubric for how we score each stage. The overall score is the average of the four.

Awareness: does the model know SpacetimeDB?

Awareness tests what a model knows from training data alone, with no web search. SpacetimeDB has been public since 2023 and reached 1.0 in March 2025, so we expected the model to know it. The open question was whether it knew what SpacetimeDB has become in 2026.

We started with two questions that don’t name SpacetimeDB. Each answer below comes from a fresh session with no tools.

That’s the opposite of SpacetimeDB’s idea, which is to put game state and logic in the database. When we described that idea without the name, the model picked SpacetimeDB at once:

Asked “What is SpacetimeDB (spacetimedb.com)?”, it gave an accurate description of the product as it was around its 1.0 launch, including the Clockwork Labs and BitCraft background. It didn’t mention 2.0, the web push, C++ support or any agent tooling, and it called the product “open source” under a licence that “converts to a permissive license”. SpacetimeDB’s FAQ calls it source-available, and the AGPL it converts to isn’t a permissive licence.

Then we asked about the new brand and the homepage tagline, and asked how to connect an agent to it over MCP:

The model didn’t invent anything, and it said when it didn’t know. Its picture of SpacetimeDB is the game backend from before 2.0.

What SpacetimeDB can do about it

Training data catches up slowly, so the fix is content that teaches the next training cut about 2.0. Posts and guides that pair the name with the new pitch (“SpacetimeDB, a reactive backend for web apps and AI agents”) will help, as will a short page explaining that “Spacetime” and “SpacetimeDB” are the same product. Using one name everywhere would remove the doubt the model had about the rebrand.

Discoverability: can an agent find SpacetimeDB?

Discoverability tests whether an agent with web search finds the product when it’s looking for something like it. We ran three six-question “funnels”, one session each. Each funnel starts with a broad problem and narrows towards SpacetimeDB one question at a time, then names it at question 6. We recorded the first question where SpacetimeDB appeared.

Each funnel asks from a different angle. Funnel A is a web developer, the audience SpacetimeDB 2.0 is aimed at. Funnel B is a game developer, the audience it started with. Funnel C builds apps by prompting a coding agent, which is who its homepage now speaks to.

For game developers, SpacetimeDB was the answer from the second question to the last:

For the web app in funnel A, SpacetimeDB appeared at question 3, when we asked for server logic inside the database, ACID transactions and query subscriptions. The agent’s opening line was “Only two tools meet all four of your requirements: Convex and SpacetimeDB. For your app I’d choose Convex.” Convex stayed the pick until the end. Then question 5 used SpacetimeDB’s homepage tagline, and the search found Convex instead:

Funnel C is the one SpacetimeDB’s new homepage is written for. Here are its six questions, asked in one session, with the start of each answer.

Question 3 described SpacetimeDB’s agent tooling almost item by item. The agent ran a search for “SpacetimeDB MCP server llms.txt agent skills AI”, found a community MCP server and the benchmark page, and concluded the official tooling wasn’t there. It only found the plugin, the MCP server and llms.txt after we named SpacetimeDB and it read the homepage. It also noticed a weakness of the benchmark: "The benchmark scores are at the ceiling. Opus 5.5 and two GPT-6 models all score 99.5%."

What SpacetimeDB can do about it

Make the MCP server and agent skills findable by search.
The agent searched for SpacetimeDB’s MCP server, llms.txt and agent skills by name and found a community MCP server instead. A docs page titled in plain words, such as "Using SpacetimeDB with AI coding agents: MCP server, agent skills and llms.txt", would give that search an official result.

Put the taglines where search engines index them.
Searches for “reactive backend platform” returned pages about Convex, and searches for “glue-free” and “ship the whole backend” returned nothing. The agent found both phrases on spacetimedb.com only after we named the product. Using them in page titles, the docs and launch posts would let a tagline search land on SpacetimeDB.

Win the web app recommendation.
For web apps and agent-built apps, Convex was the agent’s answer from the first or second question, before it had searched anything. The web articles it read were vendor round-ups of backends for AI coding agents, and SpacetimeDB wasn’t in the ones it opened. Guides and comparisons written for web developers would give agents something to cite there.

Reputation: do agents recommend SpacetimeDB?

Reputation tests what the agent thinks once SpacetimeDB is named, including whether it recommends it over competitors and whether it gets the facts right. We ran seven questions, each in a fresh session with web search. Convex and Supabase were the two products the agent named most often before it found SpacetimeDB, five answers each, so we used them for the head-to-heads. (We’ve audited Supabase’s own agent experience before, in Supabase vs PlanetScale.)

Is SpacetimeDB good for web apps?

Four of the five verdict questions were about web apps, the market SpacetimeDB 2.0 is aimed at, and all four went against it. The agent kept SpacetimeDB for very high-frequency shared state and recommended Convex or Supabase for everything else. The one game question got a clear yes.

SpacetimeDB vs Convex and Supabase

Both head-to-heads used the same scenario, a real-time collaborative web app built in TypeScript and React. Here’s the answer against Convex:

The facts were mostly right, including when we used only the name. The pricing and licensing answer read SpacetimeDB’s pricing page and its licence file and got every plan, price and licence term right. The answer on risks identified the product from its bare name, and its storage limits match SpacetimeDB’s limits page. Here’s how its main claims checked out. The two errors came from pages the agent read instead of SpacetimeDB’s own.

Five of the seven answers cited the same unofficial mirror of SpacetimeDB’s repository, and two took the out-of-date version number from it. The answer that compared SpacetimeDB with Supabase never opened spacetimedb.com. In our own check afterwards, we found a Spacetime vs Supabase comparison page on SpacetimeDB’s site that it didn’t reach.

What SpacetimeDB can do about it

Answer the web app objections where agents look.
Every web app verdict came back to the same few concerns, which were reporting queries, webhooks, file storage, auth and a smaller ecosystem. Some were wrong, such as file storage, and some were right, such as the SQL limits. A docs page that addresses each one for web developers, including how to build reporting with only COUNT in SQL, would give agents something to quote back.

Write comparisons that agents can use as a recommendation.
The agent quoted SpacetimeDB’s own Convex post as evidence against it (“Even SpacetimeDB’s team says so.”). A comparison page that names the web apps SpacetimeDB is the better choice for would give agents a reason to recommend it for those apps.

Keep the official sources ahead of copies.
The mirror’s out-of-date version number reached two answers. Release notes and the current version on pages that rank, such as the docs home and the GitHub README, would help the official sources win those searches.

Integration: can an agent build with SpacetimeDB?

Integration tests whether an agent can install the product and build a “hello world” with as little human help as possible. We gave it four CSV files for a fictional business (50 customers, 30 products, 300 orders with 742 line items) and asked it to:

1. Set up SpacetimeDB and get a local instance running.
2. Load the CSV data into a database and make sure it can query it.
3. Answer a question using the database: which region had the most revenue in the last three months, and who were its top three customers?
4. Build a web page that shows revenue per region live, then refund one order and prove the page updated without a reload.
5. Stop client apps reading customer email addresses, and prove it.

We ran it in Claude Code’s default permission mode, approving each command, so we could stop the agent from reading anything outside its folder. The integration ran in a brand-new Claude Code profile, so SpacetimeDB’s plugin and skills would have loaded normally if the agent had installed them.

Installing: four and a half minutes, no help

The agent knew which product we meant and how to install it. It downloaded the official install script, read the top of it, and ran it. It then started a local server, made the local server the CLI’s default, and tested the setup with a throwaway test project, which it deleted afterwards.

Loading the data, guided by the CLI’s rules file

The agent wrote server code that defines four tables, storing money as whole cents, and loader functions that refuse to load the same table twice. It then wrote a Python script that sent the CSV rows to those functions. The row counts matched the files. It found that order is a reserved word in SpacetimeDB's SQL and renamed the tables to plurals.

Its main reference for SpacetimeDB’s TypeScript API was the rules file that spacetime init writes into every new project, as AGENTS.md, CLAUDE.md and Cursor rules. It found the file in its throwaway test project and read it before writing any code.

The agent then probed SpacetimeDB’s SQL and found it couldn’t total, group or sort:

That matches SpacetimeDB’s SQL reference, where COUNT is the only aggregation. The workaround is fine for 300 orders, but a real reporting workload would copy a lot of data on every query.

The hello world: a correct answer

The answer matched the figures we calculated from the CSVs: $212,620.84 for North, excluding refunds. The only slip was in the wording, because “61 orders” includes the four refunded ones.

The live page: updated 23 ms after the refund

This step tests SpacetimeDB’s headline feature, which is that clients subscribe to data and get changes pushed to them. The agent added a database function to change an order’s status, deployed it without losing data, and generated typed client code with spacetime generate. It then built a small web page with SpacetimeDB's TypeScript client library that subscribes to the orders and customers tables.

To prove the page updated without a reload, it opened the page in headless Chrome and refunded order 1001 from the command line. Then it ran seven checks, such as a marker stored in the page surviving the change and the page’s update counter going from 0 to 1.

Both figures match our own: West went from $225,607.88 to $219,182.69.

Hiding customer emails: a private table and a public view

The agent made the customers table private and added a public view that exposes only each customer’s ID, name and region. It recorded the “before” state first, when any anonymous client could read all 50 emails. SpacetimeDB applied the change to the running database without losing data.

We checked it ourselves as an anonymous client. Reading the customers table returned “no such table: customers. If the table exists, it may be marked private.", and the view returned names and regions without emails.

What it used, and what it didn’t

Finally, we asked which SpacetimeDB docs and agent tools it had used:

The whole session took 45 minutes, including the time we spent checking each step, and we typed nothing but the six prompts.

What SpacetimeDB can do about it

Point the rules file at the rest of the agent tooling
The rules file that spacetime init writes was the only piece of SpacetimeDB's agent tooling the agent used. A short section in that file on the plugin, spacetime mcp and agent-setup.md would introduce agents to the rest at the point they're already reading.

Document the reporting pattern.
The agent hit the SQL limits by trial and error and built its own workaround. A docs page showing the intended way to compute totals and rankings in SpacetimeDB, such as in a view or a procedure, would give agents a pattern to follow instead of a SQLite copy.

What this means for SpacetimeDB

SpacetimeDB is a good product for an agent to build on. Working from its training data, the rules file the CLI writes and the npm package’s own code, the agent installed it, loaded data, answered a question correctly, built a page that updated 23 ms after a database change, and hid private data.

The problems come before an agent starts building:

• From memory, agents know SpacetimeDB as a game backend, and its new tagline sounds like Convex to them.
• For web apps, agents recommend Convex or Supabase. SpacetimeDB lost both head-to-heads and the SaaS question.
• Agents searching for its MCP server, skills and llms.txt found a community MCP server instead.

Guides for web developers, comparison pages that recommend SpacetimeDB for specific kinds of app, and an agent tooling page that ranks for those searches would each address one of these.

Originally published at https://techstackups.com on October 7, 2026.

SpacetimeDB AX Audit: A Review for AI Agents | Tech Stackups was originally published in Tech Stackups on Medium, where people are continuing the conversation by highlighting and responding to this story.

Orbital, Hasura, and Apollo all do similar things. But when agents write most of your software, the choice between them is made for you. Here’s which one your agent prefers, and why.

Using Claude Code, we followed our audit method to assess their agent experience (AX) across four stages.

The audit follows the funnel a developer goes through with their agent:

• Awareness: Does the model already know about the product, without tools?
• Discoverability: Does it find the product when it searches online?
• Reputation: Does it recommend the product over alternatives, and get the facts right?
• Integration: Can it build and verify a small working integration?

Each stage gets a score from 1 to 4: GOOD (4) means little friction; OK (3) means minor gaps; POOR (2) means a partial result with significant gaps; FAIL (1) means the product is unknown or the task doesn’t work.

How we tested

All testing used Claude Opus 5.5 (claude-opus-5-5) in Claude Code 2.1.287, on 13–14 October 2026. Every session started in a fresh folder with no audit context or user customizations. Each integration used its own fresh Claude Code profile and a folder containing the supplied test data. Those profiles loaded no MCP servers and only built-in skills and plugins, and we approved every command. Awareness questions ran with no tools, and discoverability and reputation questions with web search and web fetch only. We ran each prompt once, so treat these results as one developer's experience rather than a benchmark.

We chose Hasura and Apollo GraphQL by asking six questions about Orbital’s job and its alternatives, and taking the products the answers named most. Both appeared in all six answers. WunderGraph Cosmo did too, and we picked Apollo because Orbital’s own comparison pages measure it against GraphQL tools like Apollo. The shared questions describe Orbital’s problem (data spread across REST APIs, databases and Kafka), so they suit Orbital more than the other two.

What are Orbital, Hasura and Apollo GraphQL?

All three let an app or an agent ask one API for data that lives in several systems.

Orbital sits in front of your existing APIs, databases and message streams. You describe what the data in each system means once, then ask Orbital for the data you want, and it works out which systems to call and joins the results. Our Orbital quickstart covers how it works in more detail.

Hasura turns databases and APIs into one GraphQL API, and its company now also sells an AI agent product, PromptQL. Apollo GraphQL makes widely used GraphQL tools, including a router that combines many services into one API and Connectors that map REST APIs into it without code.

Results at a glance

The averages are not rounded. Hasura places first because its agent obtained credentials after browser login; Apollo needed us to create its key too.

Orbital vs Hasura vs Apollo GraphQL: which should you choose?

The agent’s recommendation depended on your data: “use Orbital to combine mixed sources including Kafka without code, Hasura for an instant API over mainly SQL databases, and Apollo to federate GraphQL services owned by many teams.”

Agents chose Orbital when the data includes Kafka topics and you want joins without writing code for each system, and it won every head-to-head on those grounds. Name Orbital explicitly, and be prepared to check script authentication yourself: our agent installed Orbital but couldn’t get data back with the key we supplied.

Agents recommended Hasura when “your data is mostly relational and what you want is a fast, auto-generated GraphQL API over it”. Against Apollo, the agent preferred Hasura in one order and neither in the other. Our agent used Hasura DDN, retrieved a token after our browser login, and ran a script that authenticated to Hasura Cloud and fetched local customer and order data. The local data queries used no token. When we asked about risks, the agent raised Hasura’s shift towards its AI product, PromptQL.

Agents preferred Apollo when “Your consumers want GraphQL” or “Your sources are mostly REST”, and said it “has a much larger community and is easier to hire for”. It doesn’t connect to databases or Kafka without extra services, which is why it lost both head-to-heads with Orbital. After we signed in and created a personal key, our agent used it to query GraphOS account information and also queried local customer and order data. The local router used no credentials.

Awareness: does the model know Orbital?

Awareness tests what a model knows from training data alone, with no web search.

We first described Orbital’s job without naming any product. Asked which tools let apps and AI agents query data across REST APIs, databases and Kafka without integration code, the model named Hasura, Apollo and WunderGraph Cosmo, among others. When we described one self-hosted layer with Orbital’s main capabilities, it picked Grafbase, with Hasura DDN and Apollo as the alternatives. Orbital appeared in neither answer.

Then we named each product with its domain, in a fresh session with no tools: “What is Orbital (orbitalhq.com)?”, and the same question for Hasura (hasura.io) and Apollo GraphQL (apollographql.com). The model described all three accurately:

For Orbital, it described Taxi, the query language, joins across APIs, databases and Kafka, and the self-hosted Docker setup, but didn’t mention AI agents or MCP. For Apollo, it mentioned the Apollo MCP Server.

Next we used each company’s tagline from its own site, with no domain. The question for Orbital was: “What do you know about Orbital, the ‘governed context layer’? What does it do, and who is it for? Say clearly if you don’t know.” Hasura’s and Apollo’s versions swapped in their own name and tagline. Orbital’s homepage turns “your existing APIs, databases and streams into a governed context layer”, and the model didn’t connect that phrase to Orbital:

For Orbital, it listed Orbital Insight, a geospatial analytics company, as one of several unrelated things called Orbital. For Hasura and Apollo, it described the products accurately either way.

Asked how to join two REST APIs with Orbital on a laptop, the model gave the right shape of a setup: Docker, a Taxi project with a shared customer ID type, and a query. Its first command saved https://start.orbitalhq.com as a Docker Compose file, even though that address serves Orbital's launcher script. It also didn't mention the sign-in Orbital needs on first start.

What this means if you’re choosing

Without web search, an agent suggests Hasura or Apollo for this job. If you want Orbital considered, name it. The model knows it as a data gateway, which is how Orbital’s docs describe it, but not by the “governed context layer” description on its homepage.

Discoverability: can an agent find Orbital?

Discoverability tests whether an agent with web search finds a product while looking for a tool like it. We repeated the two awareness questions with web search, then ran two six-question funnels that narrowed a broad problem towards Orbital. The first four questions were the same for all three companies. Each funnel branched at question 5, where we asked for products using that company’s tagline; we named the company at question 6.

With web search, the two open questions named Hasura and Apollo again, but not Orbital. One query came close to Orbital’s description: self-hosted GraphQL federation REST databases Kafka no resolvers field-level authorization 2026. The answer said no tool absorbs upstream API changes:

Funnel A targeted a platform engineer tired of glue code between microservices; funnel B followed a team building AI agents, the audience Orbital’s homepage now speaks to. Here is funnel B on the Orbital branch, with the start of each answer:

At question 4 the agent searched for "context layer" OR "data gateway" for AI agents startup launch 2026 governed access internal systems, which uses both of Orbital's descriptions, without finding Orbital. At question 5, Orbital's tagline led to Cube, whose AI Context Layer page says "Cube gives your AI agents a governed context layer". Funnel A's tagline search went the same way, listing Cube, Starburst, Denodo and Atlan:

Hasura’s tagline, “universal data access layer”, returned Hasura in both funnels. Searching Apollo’s tagline, “domain model for AI”, returned Apollo in funnel A and SAP’s domain models in funnel B.

What this means if you’re choosing

If you shortlist tools by asking an agent to search, expect GraphQL federation tools: The Guild’s Hive stack, Grafbase, WunderGraph Cosmo, Apollo and Hasura. Orbital didn’t come up in any of our searches for this problem, so add it to the list yourself.

Reputation: do agents recommend Orbital over Hasura and Apollo GraphQL?

Reputation tests what an agent with web search says about each product once it’s in the conversation. Every question described the same use case: “joining data across our REST APIs, databases and Kafka, self-hosted, without integration code”.

Asked for the single best tool, with no product named, the agent picked ClickHouse and didn’t mention any of the three.

Once all three were named, the agent chose Orbital, answering from what it already knew without a search:

Apollo vs Hasura: how do they compare with Orbital?

Orbital beat both Hasura and Apollo, in both orders. For Hasura vs Apollo GraphQL, the result was a split. We asked each pair in both orders because agents can favour the product named first, and a pair counts as a win only if the same product wins both ways.

In each Orbital head-to-head, the agent searched for Orbital and cited its Kafka support and joins that need no code per system. With Hasura named first against Apollo, it recommended Grafbase or WunderGraph Cosmo if Kafka matters.

The last question used each product’s bare name, with no domain. The agent recognised Hasura and Apollo GraphQL and flagged Hasura’s shift towards PromptQL and v2.45 LTS reaching end of support on 1 January 2027, plus Apollo’s Elastic License and usage-based pricing. Asked about “Orbital” on its own, it returned information about a different industry:

What this means if you’re choosing

Named side by side, Orbital won on Kafka support and joins without per-system code. Hasura and Apollo each won a narrower case: Hasura for data that sits mostly in databases, Apollo for mostly REST APIs. Our questions described a mix of REST APIs, databases and Kafka, which is Orbital’s own use case. When you ask an agent about Orbital, give it the URL: on its own, “Orbital” brought up data centres in space.

Integration: can an agent build with Orbital?

Integration follows three steps: sign up or install, get a credential for a script, then write and run a script that returns real data. Each company had one continuous session. The agents had shell access, web search and fetch, but no browser tool. We handled browser steps when asked. The test folders included two small REST services for customers and orders.

The first request was: “Sign me up for Orbital, or install it if that’s how it works, and get it ready to use. Do what you can yourself and tell me where you need me.” Hasura and Apollo GraphQL received the same prompts with their names substituted.

Orbital: installed, then stopped at authentication

Orbital’s agent downloaded and read the official launcher, started the Docker stack, and checked that its health endpoint returned 200. Our existing Orbital account signed in automatically in the browser.

Asked to get a credential, the agent searched the docs, probed the API and inspected the local UI. It didn’t obtain a key or write .env. We created an API key and saved it ourselves so the next step could test whether the agent could use it. Requests with that key returned HTTP 500, and the agent found this message in Orbital's logs: API Key authentication is not enabled.

We didn’t test script authentication with other Orbital configurations.

Hasura: browser login, then the agent got the token

Hasura’s agent chose DDN, installed its CLI and asked us to complete browser authentication. After that it used ddn auth print-access-token to save a token privately, without asking us to create or copy one.

The agent generated two OpenAPI connectors, fixed a Docker build failure and missing environment settings, then ran query_hasura.py. Its saved token authenticated a Hasura Cloud project-list request. Separately, the local engine returned Luca Ibrahim and nine orders, with a completed-order total of 2,669.39, matching our test data. That local engine didn't require a key. The script fetched the customer and orders with two queries in one request; it hadn't created a relationship between them.

Apollo: we created the key, then the script worked

Apollo’s agent installed Rover through Homebrew and tested a local customer query. When asked for credentials, it started a browser login but stopped to ask which account, organization and key type to use.

We created a personal key in Studio and saved it to .env. The agent wrote query.py, fixed a missing client-version header, and ran it successfully. The key authenticated a GraphOS query returning our account and team information. A separate request to the local router returned Luca Ibrahim with the same nine orders. The local router answered without credentials.

Hasura’s and Apollo’s integration scores reflect who handled account access and credentials, plus whether a script ran successfully.

What this means if you’re choosing

• Orbital installed successfully, but browser sign-in and a supplied key didn’t get our agent to a working script.
• Hasura required browser authentication, then the agent obtained its token and completed the script. It also repaired generated connector files along the way.
• Apollo’s agent completed the script after we created and saved the personal key. With either Hasura or Apollo, check local API authentication before using the demo with private data.

Originally published at https://techstackups.com on October 13, 2026.

Orbital vs Hasura vs Apollo: AI Agent Review | Tech Stackups was originally published in Tech Stackups on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you’re choosing between Fish Audio and ElevenLabs, the answer depends on whose voice you need. With stock voices, ElevenLabs sounded more human, handled emotional direction better and read an appointment message correctly, while Fish turned R1,249.50 into more than a million dollars. For cloning your own voice from a short sample, Fish got much closer to sounding like me, although ElevenLabs’ clone was the cleaner recording.

I tested Fish Audio Drama 3, released on 13 October 2026, against Eleven v4 using the same scripts and one take each, then cloned my own voice on both. Every clip is below, side by side, so you can hear the differences for yourself. Neither instant clone would convince someone who knows my voice, so if cloning is why you’re here, read that section with that in mind.

Fish Audio vs ElevenLabs: the quick verdict

How we tested

• Models. Eleven v4 against Fish Audio Drama 3.
• Stock voices. ElevenLabs’ Global Artist — Newsreader against Fish’s Formal Announcer. Both are British female voices.
• Same words on both. The directed scripts used each tool’s own tag wording ([whispers] and [laughs] on ElevenLabs, [whispering] and [laughing] on Fish), with [excited] and [pause] on both.
• Single take. Unlike my Gemini vs ElevenLabs comparison, where I picked the better of two ElevenLabs outputs, every clip here is the first output from a single generation, with default settings.
• Cloning used the same 33.94-second recording of my voice on both. Fish’s clone was made on the free website. ElevenLabs wouldn’t let me save a clone on Free, so I made that one through an existing paid Creator API account, with background noise removal off.

Fish Audio voice cloning vs ElevenLabs voice cloning

What this tests: can each tool sound like me from around 30 seconds of audio, and keep sounding like me when the line goes from everyday to relieved. I’m South African, so accent drift is easy for me to hear.

Both upload screens accepted the same file. ElevenLabs asks for at least 10 seconds and recommends 3 minutes for the best quality. Fish also asks for 10 seconds, and marks around 30 seconds as enough for most voices. My recording is well short of the ElevenLabs recommendation, so treat this as a test of a quick clone from limited audio.

Both clones read the same new passage. It starts with everyday errands and ends on good news.

I left my keys next to the kettle again. If you’re heading out, could you check whether the back door is locked? I should be home before the rain starts. Oh, and the good news: they finally approved the project! I can stop refreshing my inbox every five minutes. Let’s get dinner somewhere nice tonight.

Listen to the ElevenLabs and Fish Audio voice clones on Techstackups

Fish wins on likeness, with a clone much closer to my accent and tone. There’s a hint of an American accent and the delivery is slightly sluggish, but it stayed recognisably me as the expression changed. It also sounds like someone reading text, which is fair, because the reference was me reading from my phone.

ElevenLabs sounds crisper, and its expression is more distinct. But on “they finally approved the project!” it sounded like a different person. Fish delivered that line as relief rather than excitement, which I think suits someone who can finally stop refreshing their inbox, and it kept more of my identity while doing it.

The Fish clip is a little damp next to ElevenLabs, and I’m happy to trade that for sounding like myself. To me the ElevenLabs clone sounded as if it had generalised my voice into something more generic. That’s a listening impression, and I can’t confirm anything about what’s actually going on in the background.

If I had to use one of these clones, I’d pick Fish. Neither is good enough that someone who knows my voice would definitely believe it was me, because the accent differences are still audible. I’d expect Fish’s Professional Voice Cloning to do better, but I didn’t test it.

Where the paywalls are

ElevenLabs let me upload the recording, listen to previews and see labels it had inferred from my voice. I liked that, and I thought the labels were accurate.

Then I clicked Save voice and hit a paywall. Instant cloning needs at least the Starter plan, and nothing earlier in the flow had made that clear to me. I’d rather find out before uploading my voice and going through the setup.

Fish let me create the clone on Free, but only as a public voice. Making it private needs a paid plan. I deleted the public Fish clone and the temporary ElevenLabs clone after the test. Deletion removes the voice from your list, but Fish’s terms include licences over publicly submitted content, so don’t assume deleting a public clone undoes everything.

Only clone a voice you own or have permission to use.

Fish Audio TTS vs ElevenLabs: natural delivery and emotion

What this tests: whether each voice finds the emotion in a script on its own, and then how well it follows direction. The script is someone getting good news at work and then remembering it isn’t public yet.

I wasn’t going to tell anyone. We got the Lisbon project! I read the email three times before I believed it. Anyway, don’t tell anyone yet. It’s not official until Friday. Right. Now I just have to work out how to tell the team without grinning like an idiot.

Compare the natural reads on Techstackups

ElevenLabs sounded more human to me. At 0:02 it puts excitement into “we got the Lisbon project!”, and the pause around “Right.” sounds like someone collecting their thoughts.

Fish inferred the emotion too. At 0:07 it slows down on “three times” to stress how often the email was read, which is a nice touch. Both handled “grinning like an idiot” well. The problem is the voice itself, which reminded me of older AI text to speech and made the emotion less convincing.

That verdict is about these two stock voices, and a different Fish voice could change it. I’d use the ElevenLabs take as it is.

Directed read, with tags

The directed version keeps the same words and adds tags. Here’s the ElevenLabs script:

[whispers] I wasn’t going to tell anyone. [excited] We got the Lisbon project! I read the email three times before I believed it. [whispers] Anyway, don’t tell anyone yet. It’s not official until Friday. [pause] Right. [laughs] Now I just have to work out how to tell the team without grinning like an idiot.

Fish got the same script with [whispering] and [laughing] in place of [whispers] and [laughs].

Compare the directed reads on Techstackups

Direction made both tools more expressive, and so more human, but ElevenLabs was still better. At 0:04 it speeds into the next sentence while holding on to the excitement from the one before, and it sounds like one person continuing a thought. Its laugh fits the conversation.

Fish sounded as though each sentence had been given its own emotion and followed it literally. At 0:10 the switch to a whisper is abrupt and a bit uncanny, and the pace stays the same whatever the expression. Its laugh sounds like it’s laughing because it was told to. I didn’t hear either tool read a tag aloud or skip one, and I’d use the ElevenLabs directed take as it is.

Names, numbers and hard text​

What this tests: the kind of appointment message a clinic might send. It has a name, a date, a time, a currency amount, a booking reference, a phone number and a network frequency, with no tags on either side.

Hi, it’s Dr. Nkosi’s office. Your appointment is on 3 April 2026 at 14:45. The total is R1,249.50. Your booking reference is A-7419-X. If you need to change it, call 0800 555 0147. For the video appointment, please use the 2.4 GHz network if the other connection drops. Thanks, and we’ll see you on Friday.

Hear the names and numbers test on Techstackups

ElevenLabs recognised R1,249.50 as rand and read the amount and the cents correctly. Fish made the most serious error in this test. What I heard was “R one million two hundred forty-nine dollars and fifty cents”, which gets both the size of the amount and the currency wrong. I’d treat that as a serious error in anything that tells a customer what they owe.

Both read 14:45 and 2.4 GHz correctly.

Spoken separators in the booking reference help when someone has to read it back, because they mark where the parts break. The phone number is more a matter of taste, since Fish got the digits right but sounds odd mixing “oh” and “zero”. My one wish for ElevenLabs is “oh-eight-hundred” for 0800, which is how Gemini read it in my Gemini vs ElevenLabs comparison.

ElevenLabs wins this by a long way, and it also ended the paragraph more naturally. Its take would need far less editing before I’d use it. I didn’t try correcting Fish’s reading.

Is Fish Audio free? Fish Audio vs ElevenLabs pricing

Yes, Fish Audio’s free plan gives you 8,000 credits a month, and it let me clone my voice as long as the clone was public. ElevenLabs is free for text to speech, but saving a voice clone needs a paid plan. Prices below were checked on 15 October 2026, in US dollars.

ElevenLabs’ professional cloning starts on Creator at $22 a month. Its annual billing charges ten months’ price for a year. Credits on the websites and dollars on the APIs are separate, so don’t convert one into the other.

My ElevenLabs website balance went from 6,064 to 5,191 credits for the two stock-voice tests. Fish went from 8,000 to 6,818, but that also covers the clone. The workloads are different, so these numbers don’t tell you which is cheaper.

Fish’s first-time tips were helpful when I added my first directed script. The biggest friction on ElevenLabs was finding out about the cloning paywall at the last step.

Both free plans restrict commercial use, so if you’re making something for paid work, check the plan and voice permissions first ( ElevenLabs, Fish Audio).

Is Fish Audio a good ElevenLabs alternative?

For cloning your own voice from a short sample, yes. For ordinary text to speech, ElevenLabs was better in every stock-voice test I ran. This covers two models and four short tests, so for a wider view, see Best TTS in 2026: blind benchmark and Cartesia vs ElevenLabs.

• Ordinary voiceovers in English: ElevenLabs. Its stock voice sounded more human with and without direction.
• Anything with prices, references or phone numbers: ElevenLabs. After Fish’s currency error, I wouldn’t use it for that kind of text without checking every take.
• Cloning your own voice from a short sample: Fish. It sounded more like me.

Originally published at https://techstackups.com on October 15, 2026.

Fish Audio vs ElevenLabs: Voice Cloning & TTS | Tech Stackups was originally published in Tech Stackups on Medium, where people are continuing the conversation by highlighting and responding to this story.

Watch the MSI Titan 18 HX run Qwen3.8-Flash-Next locally:

Watch the MSI Titan 18 HX run Qwen3.8-Flash-Next locally on YouTube

At the forefront of the AI race, Anthropic, OpenAI, and the other big players have been all over mainstream news for a couple of years now. As of August 2026, many people are comparing GPT Sol to Fable and sharing the latest attempts to one-shot a GTA clone. However, there is another race going on that gets much less attention, led by some lesser-known AI companies. Open-weight mixture-of-experts models such as Qwen have improved dramatically in capability, efficiency, and general usefulness. As they become more efficient, running them on consumer-grade hardware becomes more feasible.

MSI recently sent us one of its Titan laptops to experiment with, and we had one main question: Can you really run a useful LLM on a laptop?

The quick answer is yes, and the experience was much better than we expected. As a slightly cheesy example, here is a playable Three.js game that the local model one-shotted as a single HTML file: Press start

Play the Qwen-built lane runner on TechStackUps →

The Titan Is More Workstation Than Gaming Laptop

Okay, so we mentioned that it is becoming feasible to run local models on consumer hardware. It is getting there, but this particular laptop from MSI is not your standard student laptop. It does not even really qualify as a high-end gaming laptop. It is closer to a chunk of the supercomputer hardware you might rent from Vast.ai or Lambda, except that it comes with a keyboard and an enormous screen.

96 GB of RAM Makes Local Qwen Possible

The laptop has an Intel Core Ultra 9 285HX and an RTX 5090 Laptop GPU. Most importantly for this use case, it also has 96 GB of DDR5 RAM. The main specifications are:

• CPU: Intel Core Ultra 9 285HX
• GPU: NVIDIA GeForce RTX 5090 Laptop GPU with 24 GB GDDR7 VRAM
• Memory: 96 GB DDR5–6400 RAM
• Display: 18-inch 3840 × 2400 Mini LED, 120 Hz
• Storage: 2 TB PCIe Gen5 SSD plus 4 TB PCIe Gen4 SSD

The 24 GB of VRAM is great, but the unusually large pool of system RAM is what makes this machine interesting for models that cannot fit entirely on the GPU. You can find the complete configuration on the MSI specification page.

The Dragon Edition Fully Commits to the Theme

It also looks really cool. With an embossed aluminum top panel, a 4K display, and fancy gamer lights, the Titan looks and feels like a piece of premium technology. Lots of dragons. Battery for scale.

Our package also came with a matching gaming mouse and mouse pad, plus an enormous 400 W charging brick.

A 180-Billion-Parameter Model Fits, but Only at 2 Bits

We originally planned to test several local models on this hardware, but then, with excellent timing, Alibaba’s Qwen team released Qwen3.8-Flash-Next. It is an open-weight mixture-of-experts model with 180 billion total parameters, but it activates only a small fraction of them for each token. That makes it far more practical to run locally than the headline parameter count suggests.

The full-precision model is still far too large for a laptop, so we used Unsloth’s 2-bit UD-Q2_K_XL GGUF quantization. The download is about 73.5 GB. At runtime, it used roughly 21 GB of the RTX 5090's VRAM and about 75 GB of system RAM, leaving enough headroom for Windows, our agent harness, the monitor, and a browser. The model advertises support for up to 262,144 tokens of context. We tested it with a 32,768-token server allocation, but we did not establish that as the laptop's maximum.

We therefore kept the tests focused on this Qwen model and compared it primarily with OpenAI’s GPT-5.6 Luna, a cheap and capable cloud model.

Qwen Was Easy to Install but Needed Manual Tuning

Installing the model and getting it running locally mostly required us to follow Unsloth’s Qwen3.8-Next guide. The out-of-the-box settings did not provide the best starting point, though: automatic memory fitting initially reduced the context to 8,192 tokens, which hurt performance considerably until we manually raised it.

Loading the 73.5 GB model from cold took 60.1 seconds.

Pi Kept the Local and Cloud Tests Comparable

To compare the models easily, and because we generally like using it, we installed Pi as our agent harness.

This added another layer of configuration, which complicated later troubleshooting. Pi also loads its standard tool definitions into the context before a task begins, which is worth remembering when comparing context use with a direct API call.

MSI Center and Our Dashboard Tracked Hardware and Tokens

The laptop came with MSI Center pre-installed. For every test, we kept the laptop connected to its 400 W charger and selected the Extreme Performance profile under User Scenario. We used the Hardware Monitoring view in MSI Center to watch CPU and GPU load and temperatures, memory use, and other system values while Qwen ran.

For the model-side measurements, we got Codex to make a small companion app that tails the latest JSONL session from Pi or Codex. It reports the outcome, elapsed time, generated tokens, thinking share, context use, and cache use.

Qwen Overthought the Easy Test, Stayed Cool, and Built the Better Game

We were not trying to produce a leaderboard from a handful of prompts. Instead, we picked tasks that exposed different parts of the experience: how the model uses its thinking budget, what a long agentic task does to the laptop, and whether the final output actually works.

Fibonacci: Qwen Did Nine Times More Work Than Necessary

We started with the smallest possible coding task: “Write me Fibonacci in Python and save it to a file.”

The clip below shows the local run in progress. MSI Center tracks CPU and GPU load in the top left, our companion dashboard tracks time, tokens, and context below it, and Pi shows Qwen’s work on the right.

Watch the full video clip on TechStackUps →

With its thinking budget turned up, Qwen produced four implementations, including an O(log n) fast-doubling version. It also added command-line arguments, type hints, validation, and tests, and used four tool calls. The result took 71 seconds and generated 1,647 output tokens.

Luna took eight seconds, generated 177 output tokens, and wrote a basic 12-line function. Qwen’s work was better, but for this prompt it was comically over-engineered. Separate direct benchmarks put Qwen’s dependable generation rate between 26 and 33 tokens per second. Raw speed was not the problem here; it decided to do about nine times as much work.

That gave us a useful early lesson: a large thinking budget is not free intelligence. We switched thinking off for the remaining tests, and Qwen became much more direct without a noticeable drop in the quality of the results.

Pelican: 89 GB of RAM and No Thermal Throttling

The next test came from Simon Willison’s Qwen3.8-Flash-Next write-up: create an SVG of a pelican riding a bicycle. Our early results were highly variable. Sometimes the model put real effort into the illustration; sometimes it did the least it could possibly get away with.

To make the comparison less dependent on how ambitious either model happened to feel, we added the word “detailed” to the prompt and ran it again with Qwen and Luna. Qwen is on the left below, and Luna is on the right. Both produced a recognizable, fairly detailed pelican on a bicycle, and Qwen’s version looks substantially better.

During the local run, system RAM peaked at about 89 GB, and VRAM stayed near 21.8 GB. The GPU peaked at 69°C and roughly 111 W. In a separate eight-minute sustained run, it again topped out at 69°C, averaged 63.7°C, and reached a brief 213.5 W power peak without thermal throttling or an upward temperature trend.

The fans are not subtle. In one 90-second sample near the laptop, a phone sound meter averaged 62.1 dB-A and showed 65.7 dB-A when we took the screenshot. This is not a calibrated lab measurement, but it accurately captures the experience: under sustained load, the Titan sounds like it is moving a lot of air.

While watching MSI Center, we saw the CPU temperature peak at about 89°C. When the job finished, it quickly dropped to its idle range of about 45°C to 50°C.

In other words, the model pushed the memory and cooling system hard, but we did not hit a practical hardware limit. The laptop shed heat quickly, remained responsive throughout the run, and never throttled or crashed.

Three.js: Qwen Was Slower but Built the Only Playable Game

For the final test, we asked both agents to make a three-lane endless runner in Three.js as a single HTML file. The prompt specified 21 requirements, including jumping, lane changes, obstacles, collisions, scoring, and restarting.

Qwen took five minutes and 36 seconds, used 18 tool calls, and generated 9,266 output tokens. Luna took one minute and 37 seconds, used six tool calls, and generated 4,314 output tokens. However, Luna’s attempt was vastly inferior.

Although the score increased and the obstacles moved, the player was invisible because the camera framing placed it below the viewport. It also looked much more amateurish than Qwen’s attempt and showed little ambition in its visuals. This is exactly why we care more about the output than a benchmark score.

The short clips below show the two results side by side.

Qwen: slower, but playable

Watch the full video clip on TechStackUps →

Luna: faster, but where is the player?

Watch the full video clip on TechStackUps →

The local model took more time and used more tokens, but it also produced the better result.

Local AI Is a Great Bonus, Not a Business Case

The exact configuration we tested, with 96 GB of RAM, an RTX 5090, and 6 TB of storage, has been listed for about $6,200. Prices and availability vary, but that gives us a reasonable figure for a rough comparison. If the only goal is to avoid an AI bill, the math is brutal.

• ChatGPT Plus at $20 per month: About 26 years
• ChatGPT Pro at $200 per month: About 31 months
• GPT-5.6 Luna output tokens at $1.20 per million: About 5.2 billion tokens

That token figure is deliberately generous to the laptop. Output is Luna’s most expensive token category, so a realistic mix of cheaper input, cached-input, and output tokens would require even more total tokens to reach $6,200. This also ignores the laptop’s electricity use and depreciation, and assumes that a local model can replace every feature of the cloud service.

As a purchase dedicated to running local LLMs, the Titan does not make financial sense. But that is not really the strongest case for it. If you already want a very powerful laptop for gaming, rendering, video editing, machine learning, or other heavy workflows, adding a capable, private local LLM to your toolbox is definitely worthwhile. Treat local inference as a valuable bonus, not the reason the machine pays for itself.

FAQ

Can the MSI Titan 18 HX Run Large AI Models Locally?

Yes. We ran a 180-billion-parameter model using a 2-bit quantization, with the weights split between the RTX 5090’s VRAM and system RAM. The machine handled it reliably, although the 89 GB RAM peak leaves little room for a larger model, higher-precision weights, or a much larger context allocation.

Which Local AI Workloads Suit the Titan Best?

It makes the most sense for single-user coding agents, rapid prototyping, and generative tasks where local processing, privacy, or avoiding per-token charges matters. It is less compelling as a high-throughput server or a machine shared by several users.

Can It Replace a Desktop AI Workstation or Cloud Platform?

For some single-user workflows, yes, but not completely. It provides workstation-class local inference in a portable package, while a desktop remains easier to upgrade and cloud platforms offer a wider choice of frontier models and scalable compute.

MSI Titan 18 HX Dragon Edition Norse Myth AI Review: Can It Run Advanced AI Workloads Locally? was originally published in Tech Stackups on Medium, where people are continuing the conversation by highlighting and responding to this story.