Skip to content

feat(ai-components): agent reply steps (reasoning, tool calls, approvals) - #3924

Merged
isekovanic merged 1 commit into
feat/ai-agent-groundworkfrom
feat/ai-reply-steps
Nov 6, 2026
Merged

isekovanic merged 1 commit into
feat/ai-agent-groundworkfrom
feat/ai-reply-steps

Conversation

@isekovanic

Copy link
Copy Markdown
Contributor

Ticket: https://linear.app/stream/issue/RN-485

Stacked on #3921.

🎯 Goal

Show an agent reply's steps: its rounds of reasoning, its tool calls, and the questions a tool asks before it runs. The views are ported from React's AI components (#3376). Effects follow the native SDKs instead.

πŸ›  Implementation details

Views (all props-only, none of them reads <Channel>)

  • AIReasoning shows a header that counts up while the agent thinks ("Thinking... 7s") and then reads "Thought for 12s" with the step's summary.
    • The panel opens while the agent thinks and follows the reasoning as it streams in, then folds once the step is done. Tapping the header opens or folds it.
    • The first step takes over the turn's thinking clock, so the time carries on from AIThinking.
  • AIThinking is the same header for a turn that hasn't shown any reasoning yet.
  • AIToolCall shows the call's title, a detail line, a status icon and its duration. The detail line covers waiting for approval, a sign-in or the device, and declined, didn't complete or cancelled.
  • AIToolApproval and AIToolApprovalCard ask the person a tool's question, such as "Share your location?", with the agent's reason and Allow / Don't Allow.
    • The question is only asked of the targeted user on the targeted install.
    • The buttons wait while the answer is sent, and a failed send lets the person answer again.
  • AIMessageParts / AIMessagePart render a reply's steps in order, with a neutral line for a kind this version can't show.
  • AIToolApproverProvider / useAIToolApprover say who answers the questions on this device.

Shared bits

  • In-progress states use our NativeShimmerView through a small internal Shimmer, which AITypingIndicatorView now uses too. Running calls use the platform ActivityIndicator. This matches Swift and Android, not React's CSS.
  • Durations come out as "0.4s" and "1m 5s" on both platforms. Hermes on iOS ignores narrow units in Intl, so we fall back the same way React does.
  • There's a new, optional messageParts section in the AI theme.
  • New strings are added to all 13 locales. The thinking header reuses the existing Thinking... key. The ar and he translations are new and could use a native speaker's look.

ExpoChatGPTSample

  • A reply renders its steps above its text.
  • Live reasoning comes from calypso's LiveReasoning, fed by message.updated, and "thinking" comes from the session's state.
  • Approvals go through session.decide.
  • The drawer lists the router's sessions.
  • The steps take the row's full width. The stand-in MessageList sizes bubbles to their content, and AIConversation replaces it later anyway.

🎨 UI Changes

Reasoning steps (folded, open, streaming), tool calls (running, waiting, done, failed, declined) and the approval card.

πŸ§ͺ Testing

  • Unit tests are ported from React's AIMessagePartViews suite, plus duration formatting, including the iOS fallback.
  • On an Android emulator against a local Vision-Agents router:
    • thinking, the reasoning steps folding and opening, and a web search with its spinner, check and duration;
    • the approval card: Share runs the call, Not now declines it, and no answer shows the call as failed after the router's 30s;
    • reasoning streaming into an open panel. Our router's models send their thinking in one piece, so this was driven with router-shaped events.
  • On the iOS simulator: a reply with a web search step and its duration.
  • To get an approval at all, the sample needs a phone-side tool, and the question has to carry the install id (/respond with client_id). calypso 0.2.3's responses.create doesn't send one. Both come with the client tool runner later in this stack.
  • Not tested: RTL and screen readers, on either platform.
  • yarn lint, yarn typecheck, yarn build and yarn test:unit pass.

β˜‘οΈ Checklist

@Stream-SDK-Bot

Copy link
Copy Markdown
Contributor

SDK Size

title develop branch diff status
js_bundle_size 2041 KB 2041 KB +213 B 🟒
ai_components_size 163 KB 196 KB +34134 B πŸ”΄

@isekovanic
isekovanic removed this pull request from stack #3942 November 6, 2026 14:16
@isekovanic
isekovanic added this pull request to stack #3949 November 6, 2026 14:16
@isekovanic
isekovanic removed this pull request from stack #3949 November 6, 2026 14:16
@isekovanic
isekovanic added this pull request to stack #3950 November 6, 2026 14:17
@isekovanic
isekovanic merged commit a326cdc into v9-release Nov 6, 2026
12 of 13 checks passed
@isekovanic
isekovanic deleted the feat/ai-reply-steps branch November 6, 2026 16:56
isekovanic added a commit that referenced this pull request Nov 6, 2026
…ll sign-in (#3926)

Ticket: https://linear.app/stream/issue/RN-485

Stacked on #3924.

## 🎯 Goal

Let a person sign in when an agent's tool needs a connector account
(Linear, GitHub, …), and give each install a stable id so the router can
ask only this device. This needs a small native module, `AIAgent`, which
works the same way as Swift's and Android's AI SDKs.

## πŸ›  Implementation details

**Native `AIAgent` TurboModule** (in both wrappers, sources in
`shared-native/`)
- `installId()` is synchronous and creates an id on first use, then
keeps it.
  - iOS stores `ios-<UUID>` in `UserDefaults`.
- Android writes `android-<UUID>` to a file in `noBackupFilesDir`, so a
device restored from a backup gets its own id. The file name and format
match the Android AI SDK.
- `openAuthSession(url, callbackScheme)` resolves with the redirect URL,
or `''` when the person closes the sign-in.
- iOS uses `ASWebAuthenticationSession`, ported from Swift's
`AIWebAuthentication`. It needs no usage description, so it's always
compiled; the podspecs only add `AuthenticationServices`.
- Android uses an Auth Tab where the app and the browser support it
(Chrome 137+). Otherwise it opens a Custom Tab and catches the redirect
through the app's scheme in `onNewIntent`. Coming back to the app
without a redirect counts as closed.
- Auth Tab needs `androidx.browser` 1.9.0, and 1.9.0 needs compileSdk 36
and AGP 8.9.1. Our minimum, RN 0.76, ships with 35. The wrappers' Gradle
files check both:
- when both are met, the wrappers build with 1.9.0 and the Auth Tab
sources;
  - otherwise they use 1.8.0 and the Custom Tab-only sources.

  `sync-shared-native.sh` copies both variants over.
- No new permission. The manifest adds a `<queries>` entry for Custom
Tabs services, which Android 11+ needs in order to find the browser.
- The codegen spec is copied into each wrapper, and
`validate-entry-isolation` keeps the copies identical.

**JS (`ai-components`)**
- `AIClientIdentity.installId()` reads the native id. If the module
isn't there, it falls back to an id kept in memory.
- `createAuthSessionAuthorizer({ callbackScheme, decline })` opens the
tool's sign-in at `url#handoff_token=…` and reads `status` from the
return. "Not now" calls `decline`.
- `AIToolAuthorization` and `AIToolAuthorizationCard`, shown under a
tool call that's waiting for a sign-in, have these states:
- Connect, then Connecting... while the sign-in is open, which stays
after a `connected` return until the step changes;
  - Not now;
  - connected or not connected;
  - expired;
  - didn't complete, or couldn't be opened.
- The theme gets a new `messageParts.authorizationCard` section.
- New strings are in all 13 locales. As before, `ar` and `he` could use
a native speaker's look.

**ExpoChatGPTSample**
- It sends the install id as the session's `client_id`, and its sessions
set a `return_url` on `io.getstream.expochatgptsample://sign-in`. The
router only accepts https or reverse-domain schemes there.
- `+native-intent.tsx` keeps Expo Router from opening that return URL as
a screen.
- A reply's row reads its message from the channel's state. The stand-in
`MessageList` doesn't re-render a row when only an attachment changes,
such as a tool call starting to wait for a sign-in.
- It hides the router's duplicate `connector_authorization` attachment.

## 🎨 UI Changes

The sign-in card under a tool call: Connect / Not now, Connecting...,
and its outcomes.

## πŸ§ͺ Testing

- Unit tests cover the launch URL, reading the return, the authorizer,
and the card's states.
- On an Android emulator (API 36), against a local Vision-Agents router
with the Linear connector:
  - the card shows;
  - Connect opens the sign-in;
  - closing it resets the card;
  - a `connected` return shows Connecting...;
  - Not now declines.

The return was sent with `adb`. I didn't sign into a real Linear
account. I ran both Gradle variants, the Auth Tab one and a forced
Custom Tab one.
- On an API 34 emulator (Chrome 113, no Auth Tab), the same flow works
through the Custom Tab.
- On the iOS simulator: the card, the `ASWebAuthenticationSession`
prompt, Cancel resetting the card, and Not now.
- On the minimum supported setup, a bare RN 0.76.9 app (compileSdk 35,
AGP 8.6) that uses the packed `stream-chat-react-native` builds against
`androidx.browser` 1.8.0. On Android 7.0 (API 24) and Android 14,
`installId()`, an app-scheme return and closing the browser all work.
- The RN CLI wrapper also builds in SampleApp (compileSdk 37, so 1.9.0).
I didn't run SampleApp itself.
- Not tested: RTL and screen readers.
- `yarn lint`, `yarn typecheck`, `yarn build` and `yarn test:unit` pass.

## β˜‘οΈ Checklist

- [x] I have signed the [Stream
CLA](https://docs.google.com/forms/d/e/1FAIpQLScFKsKkAJI7mhCr7K9rEIOpqIDThrWxuvxnwUq2XkHyG154vQ/viewform)
(required)
- [ ] PR targets the `develop` branch (stacked on #3924)
- [ ] Documentation is updated
- [ ] New code is tested in main example apps, including all possible
scenarios
  - [ ] SampleApp iOS and Android
  - [x] Expo iOS and Android
isekovanic added a commit that referenced this pull request Nov 12, 2026
Ticket: https://linear.app/stream/issue/RN-485

## 🎯 Goal

Bring the AI agent work we shipped on `v9-release` to V10: the same
components, APIs and sample apps, adapted to stream-chat 10 and the V10
i18n.

The long-press Copy/Edit menu on the person's own messages isn't here.
It isn't on v9 either, and it comes later as a V10 feature.

## πŸ›  Implementation details

The branch is one commit per v9 PR, in merge order, then the V10-only
changes, so it can be reviewed commit by commit.

**Ported from v9**
- #3920 parse agent reply steps from message attachments
- #3921 AI translations through the chat SDK, sample on the agents
router
- #3924 agent reply steps: reasoning, tool calls, approvals
- #3926 native `AIAgent` module for the install id and tool-call sign-in
- #3928 message views, reply actions and icon overrides
- #3929 `AIConversation` on LegendList
- #3931 chat options, suggestions and attachment preview fixes
- #3933 client tool runner
- #3934 polish of agent conversations
- #3936 generated UI with an A2UI renderer
- #3944 follow-up polish of agent conversations
- #3958 A2UI list-row secrets and malformed messages
- #3957 conversation state with `AIConversationController`
- #3960 `AIConversation` in its own entry point
- #3963 the crew assistant demo app
- #3974 alignment with the other SDKs
- #3975 StrictMode-safe controller lifecycle and `custom.uiAction` docs
- #3976 crew drawer layout
- #3978 the samples on `@stream-io/stream-agents` 0.4.2, in place of
calypso and their own chat layer

Where `develop` had already changed the same lines (comment cleanups,
the i18n move, a11y docs), `develop`'s version wins.

**What's different on V10**
- **Fields under `custom`.** stream-chat 10 keeps `mime_type`,
`file_size`, `ai_generated`, `generating`, `ai_ui` and the step fields
under `custom`. `parseMessageParts` and `fromStreamMessage` lift them
with a small `withCustomFields` helper, as the React SDK does.
- **Translation keys.** The 55 AI and A2UI strings use catalog keys with
their English inline, named after React's
(`aiComponents.replyActions.copy.accessibilityLabel`,
`a2ui.modal.close.label`, …), and `keys.ts` is regenerated.
`useAITranslation` returns core's typed `t`.
- **Entry isolation.** The AI entry points may reach V10's translation
context. `@legendapp/list` and `react-native-keyboard-controller` are
AI-only packages.
- **LegendList stays at 3.7.1**, the version v9 resolves. 3.7.2 keeps
fewer rows mounted after a jump, so the reused-rows test now only
compares rows that stay mounted. It passes on both versions.
- **Sample data layer.** Both samples use stream-agents' `AgentChat`,
whose stream-chat 10 reader reads `channel.messagePaginator`, activates
the channel so the client reloads it after a reconnect, and loads older
pages with `toTail()`. The samples read the reply fields from `custom`,
and the crew sample uploads voice memos with `uploadChannelFile`.
- **Sample dependencies.** Both samples depend on
`@stream-io/state-store` and `use-sync-external-store`, and pin
`@stream-io/state-store` to one copy in `tsconfig` paths. They stay on
stream-chat `^10.0.1`: stream-agents asks for `^10.0.2`, but only when
it builds its own client, and the samples hand it theirs.

## 🎨 UI Changes

None compared with v9. The screenshots are in the v9 PRs above.

## πŸ§ͺ Testing

`yarn lint`, `yarn typecheck`, `test:typecheck`, the unit tests and
`yarn build` pass, and both samples typecheck.

On an Android emulator and an iOS simulator:
- **ChatGPT sample:** reasoning, the device time tool, the generated
Lisbon plan, the drawer, reopening a conversation from history, and
starters on a new one.
- **Crew sample:**
- all eight flows, including the approval cards, the checklist and form
actions, and the agent-first ones;
- the equipment deep link, photo troubleshooting and the team lead view;
  - voice memos, recorded on the simulator with `say` and answered;
  - the clock-out form on iOS.
- **Reconnect:** the network was dropped mid-reply on both samples. The
client recovered the active channel, and the full reply appeared.
- **Older messages:** with the page size briefly set to 3, older pages
loaded until there were none left.

After the move to stream-agents, both samples were checked again on both
platforms: replies, the device time tool, the agent-first Clock in flow,
the drawer, reopening from history, handbook sources, and an approval
card into A2UI. Reconnect, older pages and voice memos were tested on
the samples' previous chat layer, not again since.

Not tested here: voice mode (the simulator's speech recognition doesn't
hear `say`), dictation, Cal.com sign-in, and charts or photos in the
ChatGPT sample.

## β˜‘οΈ Checklist

- [x] I have signed the [Stream
CLA](https://docs.google.com/forms/d/e/1FAIpQLScFKsKkAJI7mhCr7K9rEIOpqIDThrWxuvxnwUq2XkHyG154vQ/viewform)
(required)
- [x] PR targets the `develop` branch
- [x] Documentation is updated
- [ ] New code is tested in main example apps, including all possible
scenarios
  - [ ] SampleApp iOS and Android
  - [x] Expo iOS and Android
@github-actions github-actions Bot mentioned this pull request Nov 12, 2026
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants