Repository navigation
feat(agents): type a model key refusal as model_key_budget / model_key_auth, not agent_error - #695
Draft
andrewpark-scaleai wants to merge 2 commits into
Draft
andrewpark-scaleai wants to merge 2 commits into
andrewpark-scaleai wants to merge 2 commits into
Conversation
…odel_key_budget or model_key_auth Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ey_auth, not agent_error TaskError.error_type gains model_key_budget and model_key_auth. The claude-code bridge reads the CLI's api_error_type / api_error_status from its terminal result event and the A2A server passes the type to builder.failed, so the framework's PromptResponse no longer blames the agent for an out-of-budget or rejected key. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
AgentEnv should say why every run failed, and heal itself where code can. Today a run whose model key is out of budget or rejected is blamed on the agent.
When the model endpoint refuses the key, the Claude Code CLI exits 1 and its terminal
resultevent names the cause as fields:api_error_statusandapi_error_type(for example400/budget_exceededfrom LiteLLM,401/authentication_errorfrom Anthropic). The claude-code agent's A2A server ignored those fields and calledbuilder.failed("cli_error", ...)with noerror_type, so the protocol's defaultagent_errorreached the framework'sPromptResponse.error_type. Graders and dashboards then count a platform key failure as the agent's mistake, and nothing downstream can tell the two apart without matching error text.What
agentenv-framework-protocol:TaskError.error_typeis now the closed typeTaskErrorType, which addsmodel_key_budgetandmodel_key_authtoagent_errorandinfra_error. The validator and every signature that took the old two-valueLiteraluse it.resultevent into that type when the CLI fails.api_error_type == "budget_exceeded"givesmodel_key_budget. A 401 or 403, or a type ofauth_error,authentication_errororpermission_error, givesmodel_key_auth. Anything else staysagent_error. The A2A server passes it tobuilder.failed.agents/AGENTS.mddocument them.The framework side needs no change:
TerminalResponse.from_messagealready copieserror_typefrom the data part intoPromptResponse.error_type.Design decisions and tradeoffs
error_typevalues, not a new field. Every reader ofPromptResponse.error_typealready treats any value as "the turn errored", so verifiers keep skipping these turns. A later breaker or classifier can match the two key values exactly instead of matching text. The cost is that a reader comparingerror_type == "infra_error"will not match them. Nothing in this repo does._terminal_result,_status), so the mapping lives there and the server only forwards a typed value.agent_error. Rate limits (429) and bad requests (prompt too long) keep the old type. This PR only reclassifies the two causes the endpoint states outright.failed("cli_error", ...)default. I did not have their CLIs to capture what they print for a budget or auth refusal, so they are left for a follow-up rather than guessed.>=0.1.551assumes this lands in the next protocol release. If another release goes out first, bump the floor to the version that carries this change. The image installs the newest protocol at or above the floor, so the floor documents the need rather than gating it.How to verify
The test events are trimmed from captures of Claude Code CLI 2.1.325 run against a local fake endpoint that answers with each error body.
Result: 1955 passed, 1 skipped in 576s.
ruff checkon the touched files passes.Without the fix,
test_model_key_errors.pyfails for the four key cases (litellm-budget,litellm-auth,anthropic-auth,anthropic-permission) witherror_type == "agent_error", and the protocol test fails withValueError: error_type must be 'agent_error' or 'infra_error'.End to end, I ran the agent's A2A server in-process with the real
claudebinary against a local endpoint answering400 {"error": {"type": "budget_exceeded", ...}}. The terminal message carriederror_type: model_key_budget,error_code: cli_error. A404endpoint still gaveagent_error.