> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coderabbit.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Runtime verification

> Get step-by-step reproduction guides for Deep Scan findings, and run them in an isolated test environment to collect evidence.

GitHub Cloud (Runtime verification is available for GitHub Cloud repositories.) Cloud Only (This feature is available on CodeRabbit Cloud only. It is not available on self-hosted CodeRabbit deployments.) | Beta (Running guides in a test environment is in beta.)

Runtime verification helps you check whether a [Deep Scan](https://docs.coderabbit.ai/security.md#understand-deep-scan) finding is real before you decide how to address it. It has two parts:

* A **reproduction guide**: step-by-step instructions, written from the source code, that you can follow in your own staging or test environment. Guides are free and don't run code.
* **Verification**: CodeRabbit runs the guide in an isolated test environment that it prepares for the scanned commit, then keeps the result and the evidence with the finding. The agent is instructed to use only that test environment, never your deployed services.

By default, Deep Scans create reproduction guides for new or reopened Critical and High findings. You choose which findings also get a verification run, as repository defaults or for a single scan.

## Before you begin

* Runtime verification is available for GitHub Cloud repositories on CodeRabbit Cloud. If a finding's **Runtime verification** tab shows **Coming Soon**, the feature is not available for that finding.
* Viewing guides, results, and evidence requires Security read access. Generating guides, starting or canceling runs, and saving repository defaults require Security write access.
* Verification runs require an active paid CodeRabbit Agent add-on for your organization. Runs use Verify minutes, which are billed as [Agent minutes](https://docs.coderabbit.ai/code/access-and-billing.md#agent-minutes). See [Usage and billing](#usage-and-billing).
* To run a guide in CodeRabbit, the repository needs a prepared [environment](https://docs.coderabbit.ai/code/environments.md) with the dependencies and setup the application needs. You don't need one to follow a guide in your own environment.

## What happens after a Deep Scan

After a Deep Scan of the repository's code finishes, CodeRabbit looks at the open findings that the scan newly detected or reopened:

1. It generates a reproduction guide for each finding at or above the guide level.
2. When a guide is ready, it runs the guide for each finding at or above the verification level, in the repository's default environment.

Findings carried over unchanged from earlier scans don't get new guides or runs. On large scans, guides can arrive well after the scan finishes, so a completed Deep Scan doesn't mean its guides and runs have finished. Open a finding's **Runtime verification** tab to follow its progress.

If no guide level is saved for the repository, guides are generated for Critical and High findings, or for every severity selected for verification if that is broader. If the repository has no default environment ready, findings get a guide but no run.

Both levels use the same options:

| Level | Findings included |
| - | - |
| Off | None. |
| Critical | Critical findings only. |
| Critical and high | Critical and High findings. |
| Medium and higher | Critical, High, and Medium findings. |
| All | All severities, including Low. |

Verification can only run on findings that have a guide, so the verification level can't be broader than the guide level. If verification covers a severity that guides don't, CodeRabbit shows **Generate reproduction guides for every severity selected for verification.** and you can't save the defaults or start the scan. Setting verification to **Off** still generates guides. To generate neither, set both levels to **Off**.

### Set repository defaults

1. Go to **Security > Repositories**, select the repository, and open the **Settings** tab.
2. In **Reproduction and verification**, choose a level for **Generate reproduction guides** and for **Verify in a test environment**.
3. Select **Save defaults**.

The defaults apply to future Deep Scans of the repository, including scheduled and coverage-triggered scans. Saving with verification on makes you the payer for runs from those scans. See [Usage and billing](#usage-and-billing).

### Override for one scan

In the **Run deep scan** dialog, expand **Configuration**. Under **Runtime Verification**, choose levels for **Generate reproduction guides** and **Verify in a test environment**. These choices apply to that scan only and don't change the repository defaults. When the section is collapsed, it summarizes the choices, for example "Guides: High+ · Verification: Critical".

> [!WARNING]
>
> If the repository has no saved defaults, the dialog preselects **Critical and high** for guides and **Critical** for verification, even though the repository's **Settings** tab shows verification as **Off**. If you start the scan without changing them, CodeRabbit runs verification for new Critical findings and bills the runs to you. Set verification to **Off** in the dialog if you want guides only.

## Generate and use a reproduction guide

You can generate a guide for any Deep Scan vulnerability, whatever its severity.

**1. Open the finding**

Go to **Security > Findings**, open a Deep Scan vulnerability, and select the **Runtime verification** tab.

**2. Generate a guide**

In **Reproduce this finding**, select **Generate guide**. CodeRabbit first rechecks whether the finding is still present in the current source. If it isn't, the tab shows **Finding no longer present** and no guide or run is needed. You can select **Check again** later.

When the finding is still present, CodeRabbit analyzes the source and writes a guide tied to the displayed **Revision**. You can cancel with **Cancel generation**.

**3. Review the guide**

Expand **Steps to reproduce** to read the steps, the expected result, and any missing context, prerequisites, alternative methods, and limitations. Expected results are based on source analysis; they are not runtime evidence. Select **Download .md** to save the guide, or **Regenerate guide** to write a new one.

**4. Reproduce in your own environment**

Follow the guide in your staging or test environment. Check out the displayed revision, satisfy the prerequisites, and supply any environment-specific values the guide asks for. Compare what you observe with the expected result.

Generating a guide doesn't start a run in CodeRabbit.

## Run the guide in CodeRabbit

CodeRabbit can run a published guide in an isolated test environment and keep the evidence from the run. Guide execution stops after 90 minutes. Shutdown and assessment can take additional time.

**1. Prepare an environment**

Follow [Coding environments](https://docs.coderabbit.ai/code/environments.md) to choose toolchains, install dependencies, configure environment variables and startup instructions, and assign the repository. Wait until the environment and the repository assignment are ready.

If the repository has no environment, the finding's **Runtime analysis** section shows **Set up an environment to run this guide**. Select **Set up environment** to open the environment setup.

**2. Start the run**

Return to the finding's **Runtime verification** tab. Under **Runtime analysis**, in **Run this guide**, select **Run analysis**. Choose an **Environment** and select **Start analysis**. The environment you choose applies to this run only and doesn't change the repository's default environment.

**3. Follow the run**

The run moves through preparing the environment, running the steps, assessing the evidence, and saving the result. Select **Cancel run** to stop it while it is active.

To run the guide again, select **Run again**, or **Retry run** after a failed run. If the previous environment is still available, this immediately starts a new billed run in it; otherwise, CodeRabbit asks you to choose an environment. To pick a different one, open **Run options** and select **Change environment…**.

## Environment variables and evidence

A run uses the environment you choose. An automatic run uses the repository's default environment.

The agent running the guide can read the environment variables configured for that environment, including values marked as secret, and can run commands in its sandbox. Marking a variable as secret doesn't hide its value from the agent during the run.

> [!WARNING]
>
> Use a dedicated test environment for runtime verification. Don't give it production credentials or secrets the run doesn't need, and don't print secret values in test scripts or commands. CodeRabbit redacts secret values it recognizes from stored evidence, but encoded, split, or otherwise transformed values can still be saved as evidence that everyone with access to the finding can read.

CodeRabbit keeps the run's evidence with the finding: completed steps, command output, observations, and downloadable text evidence. Evidence is text only. Everyone with Security read access who can see the finding can view it.

## Interpret runtime results

The **Runtime analysis** section shows one of these results:

| Result | Meaning |
| - | - |
| Reproduced | The run produced evidence that supports the finding under the tested conditions. |
| Could not reproduce | The run did not reproduce the finding under the tested conditions. This doesn't establish that the finding is impossible in other environments or configurations. The Findings list shows this result as **Not reproduced**. |
| Inconclusive | The run could not establish a supported conclusion. Read the reason and limitations before deciding what to do next. |

**Evidence verified** appears when a finished run that isn't Inconclusive has readable evidence.

Check **Application ready** and **Target exercised** alongside the result. They show whether the run confirmed that the application was ready and that the relevant code path was exercised. A setup failure or an untested path is not evidence that a vulnerability is absent.

Under **Assessment and evidence**, review the completed steps, the retained output, **Limitations**, and **Counterevidence**. Select **View evidence** or **Download text evidence** for closer inspection. The result also shows the **Environment**, the **Tested commit**, and the **Verify minutes** used.

Results apply to the published guide and the tested commit. They can change the finding's state:

* **Reproduced** can reopen a fixed finding, unless newer activity or a dismissal takes precedence.
* **Could not reproduce** doesn't mark the finding fixed. CodeRabbit rechecks the source, and the recheck determines whether the finding still applies.
* **Inconclusive** doesn't change the finding's state.

Use the evidence to decide whether to [resolve or dismiss the finding](https://docs.coderabbit.ai/security.md#triage-and-remediate-findings), or start [Fix with AI](https://docs.coderabbit.ai/security.md#fix-with-ai).

### Filter findings by result

On the **Findings** page, the **Verification** column shows each vulnerability's latest run result or guide status. To filter, use the **Runtime verification** filter group:

* **Run result**: Reproduced, Not reproduced, Inconclusive, In progress, Failed, Canceled, or Not run.
* **Reproducer guide**: Ready, Partial, Unavailable, In progress, Failed, Canceled, or No guide.

## Usage and billing

Reproduction guides are free. Running a guide in CodeRabbit uses Verify minutes, billed at **\$0.40 per Agent minute**. You pay for the time spent running the guide and assessing the result. Each run shows its Verify minutes when it finishes.

You aren't charged for a run that fails before it publishes a result; it shows **Not billed**. Published results are charged, including **Inconclusive**. Each run is charged separately, including reruns.

Who pays for a run depends on how it started. The tab shows who started each guide and run, and who is billed for automatic runs.

* **Runs you start from a finding**: billed to you.
* **Deep Scans you start**: automatic runs are billed to you, including runs from the levels preselected in the dialog.
* **Scheduled and coverage-triggered scans**: automatic runs are billed to the person who last saved the repository defaults with verification on. If the defaults were saved with verification **Off**, or never saved, these scans generate guides only.

If the payer can't be billed, for example because the organization has no active paid Agent add-on or has reached its Agent spend cap, the guide is still published and the run shows **Not run** with the reason.

See [Access and billing](https://docs.coderabbit.ai/code/access-and-billing.md#agent-minutes) for Agent-minute pricing and spending controls.

## Troubleshoot runtime verification

When an automatic run can't start, its message begins with "Couldn't verify automatically." followed by the reason.

| What you see | What to do |
| - | - |
| **Coming Soon** on the Runtime verification tab | Runtime verification is not available for that finding or organization. It requires a GitHub Cloud repository on CodeRabbit Cloud. |
| **Generate guide** is disabled | You need Security write access. Ask an organization administrator for access. |
| **Couldn't confirm this finding is still present** | Read the recheck message and try again when the required source or context is available. No guide has been generated yet. |
| A guide lists missing context or has no runnable steps | Review its prerequisites and limitations. A guide alone doesn't run a test. |
| "This workspace has no active paid Agent add-on" | Ask a billing admin to add the CodeRabbit Agent add-on, then run the guide. |
| "Couldn't start the run. Check your QA Agent access." | CodeRabbit couldn't confirm CodeRabbit Agent access for you or your organization, for example because your GitHub account isn't linked to a workspace member. Contact [CodeRabbit support](https://docs.coderabbit.ai/support). |
| Other "Ask a billing admin" messages | The Agent add-on is turned off, an Agent invoice is unpaid, the Agent spend cap is reached, or the billed member reached their spend limit. Ask a billing admin to fix the cause the message names, then run the guide. |
| "Your current Agent minutes billing or usage limits prevent verification." | Shown in the repository defaults and the scan dialog. Ask a billing admin to review billing and usage limits before you turn on verification. |
| Any other refusal message | Follow the cause and fix that the message names. Some billing checks clear on their own; run the guide again after a few minutes. |
| "The automatic run was skipped because this repository has no default environment" | Choose an environment and run the guide from the finding, or prepare a default environment for future scans. |
| A new finding has a guide but no run | Check the verification level, whether the finding was new or reopened in that scan, the default environment, and, for scheduled and coverage-triggered scans, whether the repository defaults have a payer. |
| Inconclusive | Read the reason and the step evidence for missing dependencies, setup failures, or an unexercised target path. Address those before running again. |
| Evidence unavailable | Retry loading the evidence. Don't read missing evidence as a successful verification. |

## What's next

> [card] [Coding environments](https://docs.coderabbit.ai/code/environments.md)
>
> Prepare an environment that CodeRabbit can use to run reproduction guides.

> [card] [Triage and remediate findings](https://docs.coderabbit.ai/security.md#triage-and-remediate-findings)
>
> Use the evidence to resolve, dismiss, or fix a finding.

> [card] [Steer Deep Scans](https://docs.coderabbit.ai/security/steer-scans.md)
>
> Give future scans the repository context and security instructions they need.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.