> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# pipecat eval

> pipecat eval runs scripted and simulated behavioral evals against a running Pipecat agent, individually or as a suite from the CLI.

Run scenario-based behavioral evals. `pipecat eval run` tests scenarios, scripted or simulated, against an already-running agent; `pipecat eval suite` spawns the agents listed in a manifest and runs their scenarios concurrently; `pipecat eval judge` judges saved conversations again without an agent. All exit `0` when everything passes and `1` otherwise.

If `pipecat-ai[cli]` is a dependency of your project, run these commands with `uv run pipecat eval`. They're also available as `python -m pipecat.evals`.

Before running, `pipecat eval` loads the nearest `.env` file, walking up from the working directory, so a simulation's persona LLM and a hosted judge find their credentials the way your agent does. Variables already set in the shell win.

See the [Pipecat Evals guide](https://docs.pipecat.ai/pipecat/evals/overview.md) for concepts, the [scripted](https://docs.pipecat.ai/pipecat/evals/scripted-scenarios.md) and [simulated](https://docs.pipecat.ai/pipecat/evals/simulated-scenarios.md) scenario formats, and [manifests](https://docs.pipecat.ai/pipecat/evals/suites.md).

## eval run

Run one or more scenarios against an already-running agent, started with the dev runner (`python bot.py`). The harness connects to its `/ws` route. A simulation runs once here, whatever its `runs:` field says; a success rate over several runs is the suite's job, since each run needs a fresh agent.

**Usage:**

```shell theme={null}
pipecat eval run [OPTIONS] SCENARIOS...
```

**Arguments:**

**SCENARIOS...** `path` *required*

One or more scenario YAML files, or directories of them. A file holds one or
more scenarios under `scenarios:`, scripted or simulated, and each runs on its
own as `<file>/<scenario>`. A directory expands to its `.yaml` and `.yml`
files in filename order, non-recursively, skipping `!include` fragments (files
with no `name:`).

**Options:**

**--bot-url** `string` (default: ws\://localhost:7860/ws)

WebSocket URL of the agent: the dev runner's `/ws` route, or a Pipecat Cloud
session's. A URL without the `/ws` path is rejected.

**--verbose / -v** `flag`

Print a line for each turn and expectation as it resolves. For a simulation,
print the conversation as it happens and the judge's reasons at the end.

**--audio / -a** `flag`

Record each scenario's conversation audio (audio-mode scenarios).

**--record-dir** `string` (default: recordings)

Directory for `--audio` recordings: `<record-dir>/<scenario>.wav`, where the
scenario's `<file>/<scenario>` name has its slash written as a double
underscore.

**--cache-dir** `string`

Directory for cached synthesized user audio. Defaults to
`<user-cache-dir>/pipecat/tts`.

**--no-cache** `flag`

Disable the user-audio cache: re-synthesize every turn (no reads or writes).

**--timeout / -t** `integer` (default: 60)

Default per-expectation timeout in seconds, for scripted expectations without
their own `within_ms`. A simulation is bounded by its own `max_turns`,
`max_duration_s`, and `max_silence_s` instead.

**--logs-dir** `string` (default: .)

Directory for each scenario's logs: `<logs-dir>/<scenario>.eval.log` (plus
`.debug.log` under `--debug`, and a simulation's `.conversation.json`), named
like a recording.

**--debug / -d** `flag`

Also save `<scenario>.debug.log` with the harness's full per-pipeline logs.

**--stop-bot** `flag` *deprecated*

*Deprecated since 1.13.0.* No effect, with no replacement: the harness always
ends the agent's session after a run.

**--trigger-disconnect** `flag` *deprecated*

*Deprecated since 1.13.0.* No replacement: an agent fires
`on_client_disconnected` whenever the eval client disconnects. Only affects an
agent serving the server `EvalTransport`.

**--bot-ready-timeout** `float` (default: 60)

Seconds to wait, once connected, for the agent's RTVI `bot-ready`, which
covers the agent's startup.

## eval judge

Judge saved conversations against a simulation's or a [rubric](https://docs.pipecat.ai/pipecat/evals/simulated-scenarios.md#judging-a-production-session)'s `success:` and its judged metrics, without the agent. Each conversation gets the verdicts a run of the simulation would, with a reason for every one, passes included. Measured metrics are left out, since they come from a live run. Use it to reword a `success:` or a criterion and see the new verdicts on runs you already have. It exits `0` when every conversation passes and `1` otherwise.

**Usage:**

```shell theme={null}
pipecat eval judge [OPTIONS] SCENARIO_PATH CONVERSATIONS...
```

**Arguments:**

**SCENARIO\_PATH** `path` *required*

Scenario YAML file holding the simulation or rubric to judge with.

**CONVERSATIONS...** `path` *required*

Conversation files to judge: a run's `<stem>.conversation.json`, or the same
format saved from a real call.

**Options:**

**--scenario / -s** `string`

The simulation or rubric to judge with when the file holds several, with or
without the file's name (`book_table/available` or `available`).

**--output / -o** `path`

JSONL file to write, one line per conversation file. Defaults to
`judged.jsonl`.

**--concurrency / -c** `integer` (default: 4)

How many conversations to judge at once.

**--verbose / -v** `flag`

Print the judge's reason and each metric's score.

## eval suite

Spawn the agents in a manifest and run their scenarios concurrently. A manifest lists scenario files, and each file contributes one run per scenario it holds, scripted and simulated side by side, named `<file>/<scenario>`. A simulation runs as many times as its `runs:` field says, and every run must pass. Everything except the `suite:` list can be set in the manifest or overridden on the command line (the command line wins).

**Usage:**

```shell theme={null}
pipecat eval suite [OPTIONS] MANIFEST_PATH
```

**Arguments:**

**MANIFEST\_PATH** `path` *required*

Manifest YAML listing agents and their scenarios.

**Options:**

**--pattern / -p** `string`

Only run entries whose `name:` or bot path contains this substring.

**--scenario / -s** `string`

Only run this scenario: its full `<file>/<scenario>` name, or either half of
it, so a file's name selects every scenario it holds.

**--kind / -k** `string`

Only run scenarios of this kind: `script` or `simulation`.

**--name / -n** `string`

Run subdirectory name under `runs_dir`. Defaults to a timestamp.

**--runs-dir** `path`

Output base, overriding the manifest's `runs_dir`. A `<name>/` subdirectory
with `logs/` and `recordings/` is created under it. Defaults to `eval-runs`.

**--bots-dir** `path`

Override the manifest's `bots_dir` (bot paths are relative to it).

**--scenarios-dir** `path`

Override the manifest's `scenarios_dir`.

**--concurrency / -c** `integer`

Override the manifest's `concurrency` (how many runs execute at once). The
suite keeps that many runs going, taking the next in manifest order. An entry
that sets its own `concurrency:` never has more than that many in flight.

**--repeat / -r** `integer`

Override the manifest's `repeat`, how many times to run each (bot, scenario)
pair, to measure flakiness. Must be at least 1. A repeat set here or in the
manifest also runs each simulation this many times instead of its own `runs:`
(`1` included). Attempts run attempt-major across bots (every entry's first
attempt before any entry's second). A repeated sweep reports a pass rate per
pair and always exits `0`, since a rate is neither a pass nor a fail.

**--base-port** `integer`

Override the manifest's `base_port` (default `7900`). Each run gets \`base\_port

* index\`.

**--cache-dir** `string`

Override the manifest's `cache_dir` for cached synthesized user audio.

**--no-cache** `flag`

Disable the user-audio cache: re-synthesize every turn (no reads or writes).

**--timeout / -t** `integer` (default: 60)

Default per-expectation timeout in seconds, for scripted expectations without
their own `within_ms`.

**--spawn** `string`

Override the manifest's spawn template. Default: `"{python} {bot} --port   {port}"`.

**--python** `string`

Override the Python interpreter used to spawn each agent.

**--audio / -a** `flag`

Record conversation audio.

**--debug / -d** `flag`

Also save `<run>.debug.log` with the harness's full per-pipeline logs.

## Examples

```shell theme={null}
# Run one scenario against a running agent
pipecat eval run scenarios/capital_question.yaml

# Run every scenario in a directory, verbosely
pipecat eval run scenarios/ -v

# Run a simulation once and watch the conversation
pipecat eval run scenarios/simulated/book_table.yaml -v

# Judge saved conversations again, after rewording the success
pipecat eval judge scenarios/simulated/book_table.yaml -s available logs/*.conversation.json -o judged.jsonl

# Run a full suite
pipecat eval suite manifest.yaml

# Only the simulations, each as many times as its file says
pipecat eval suite manifest.yaml -k simulation

# Every scenario in the book_table file, then just one of them
pipecat eval suite manifest.yaml -s book_table
pipecat eval suite manifest.yaml -s book_table/available

# Only the support agent, 8 runs at a time, named output dir
pipecat eval suite manifest.yaml -p support -c 8 -n nightly
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.