Skip to content

feat(ai-components): chat options, suggestions and attachment preview fixes - #3931

Merged
isekovanic merged 1 commit into
feat/ai-conversationfrom
feat/ai-composer-extras
Nov 6, 2026
Merged

isekovanic merged 1 commit into
feat/ai-conversationfrom
feat/ai-composer-extras

Conversation

@isekovanic

Copy link
Copy Markdown
Contributor

Ticket: https://linear.app/stream/issue/RN-485

Stacked on #3929.

🎯 Goal

The composer extras of StreamChatAI's AIComposerView and React's AIMessageComposer (#3376): chat options such as "Think harder", the chip of the one picked, and starter suggestions for an empty conversation.

πŸ›  Implementation details

Chat options (message-composer/ChatOptions.tsx)

  • AIChatOption is React's type: id, title, and optional description, icon, shortTitle and customData.
  • ComposerView takes chatOptions and lists them in its sheet under Camera and Photos, as Swift's picker does. Picking one closes the sheet; picking the picked one again removes it.
  • The picked option shows as a chip above the input (SelectedChatOptionChip, Swift's look), and is sent with each message as chatOption in onSendMessage. Like Swift and Juno, it stays picked after sending, until the person removes it.
  • It lives in the composer's state store as chatOption, so an app that passes its own store can read it. useComposerChatOption reads it inside the composer.
  • ChatOptionsList and SelectedChatOptionChip are exported for custom composers.

AISuggestions (suggestions/)

  • React's API: suggestions and onSelect. A row of chips scrolling sideways, two lines each at most 160 wide, as Swift's SuggestionsView. Where it goes is up to the app.

Attachment preview

  • The remove button has a label and a bigger hit area.
  • An attachment that isn't an image shows as a file card rather than an empty Image, as React's preview does. The built-in pickers only pick photos, so this shows only with a custom mediaPickerService.

Also

  • chatOptions and suggestions theme sections, with Swift's colours from the shared tokens. Sizes and spacing come from the tokens.
  • Two strings in all 13 locales. ar and he could use a native speaker's look.

ExpoChatGPTSample

  • The placeholder sheet options (which only showed an alert) are gone. In their place is "Think harder", which asks the router with reasoningEffort: 'high', as Juno does.
  • Starters show above the composer on a new conversation, until its first question. A starter asks with the option picked in the composer, which Juno doesn't do.

🎨 UI Changes

The sheet with "Think harder" under Camera and Photos, the "Think" chip above the input, and the starters above the composer on a new conversation.

πŸ§ͺ Testing

  • Unit tests: the option is sent and stays picked, removing it from the chip and by picking it again, a non-image attachment shown as a file and removed, and the suggestions.
  • On the iOS simulator and an Android emulator against a local Vision-Agents router: picking the option, the chip, sending typed questions and starters with it picked and removed, and removing it with the keyboard up. A temporary log confirmed the sample sent thinkHarder: true with the chip and false without.
  • Not tested: TalkBack/VoiceOver and RTL, and the file card on device (no picker gives one).
  • yarn lint, yarn typecheck, yarn build and the unit tests pass.

β˜‘οΈ Checklist

@isekovanic isekovanic mentioned this pull request Nov 4, 2026
2 of 6 tasks
@isekovanic
isekovanic added this pull request to stack #3942 November 5, 2026 16:56
@isekovanic
isekovanic force-pushed the feat/ai-composer-extras branch from eb380d4 to ff67ab5 Compare November 5, 2026 16:57
@Stream-SDK-Bot

Copy link
Copy Markdown
Contributor

SDK Size

title develop branch diff status
js_bundle_size 2041 KB 2041 KB +213 B 🟒
ai_components_size 163 KB 260 KB +99463 B πŸ”΄

@isekovanic
isekovanic removed this pull request from stack #3942 November 6, 2026 14:16
@isekovanic
isekovanic added this pull request to stack #3949 November 6, 2026 14:16
@isekovanic
isekovanic removed this pull request from stack #3949 November 6, 2026 14:16
@isekovanic
isekovanic added this pull request to stack #3950 November 6, 2026 14:17
@isekovanic
isekovanic merged commit 308ef1f into v9-release Nov 6, 2026
13 checks passed
@isekovanic
isekovanic deleted the feat/ai-composer-extras branch November 6, 2026 16:56
isekovanic added a commit that referenced this pull request Nov 6, 2026
Ticket: https://linear.app/stream/issue/RN-485

Stacked on #3931.

## 🎯 Goal

Let an AI agent call tools that run on the person's phone, as
StreamChatAI's `AIClientToolRunner` does. React has no runner, so this
ports Swift's.

## πŸ›  Implementation details

**`AIClientToolRunner`** (`tool-calls/AIClientToolRunner.ts`)
- Swift's types: `AIClientTool` (a definition, optional instructions,
and `run`), `AIClientToolDefinition` (MCP's tool JSON: `name`,
`description`, `inputSchema`, `annotations.readOnlyHint`),
`AIClientToolResult` (`output`, `summary`, `failure`), with
`completedToolResult` and `failedToolResult`.
- `run(parts, send)` takes a reply's steps as they update. It runs a
call when it is `awaiting_client` for this person and this install
(`isAwaitingClient`), the runner has a tool of that name, and the call
hasn't run here before. The result goes to the app's `send`.
- A result that couldn't be sent is offered again on a later update
without running the tool again, up to `maxAttempts` (3). A tool that
throws is reported as failed. The first tool of a name wins.
- `registrations` describes the tools to the agent, sorted by name, as
Swift's do.

**ExpoChatGPTSample**
- One phone tool, `get_local_time` (the phone's clock and time zone),
shown on the reply as "Checking your phone's time".
- The tool is declared on the router session (calypso `Tools`,
`executor: 'client'`), and the session's socket answers the router with
what the runner reported.
- The router addresses a call to an install only when the question names
it, and only its `respond` endpoint takes `client_id`. So plain
questions go through `respond` with this install's id, and the runner
runs their calls from the reply.
- `responses` (calypso's `responses.create`) is the only one that takes
photos and a reasoning effort, and it takes no install. A question with
either goes there, and the socket runs the tool itself, since no step
will name this phone.

## 🎨 UI Changes

None in the SDK. The sample shows the phone tool's step, "Waiting for
the device" and then done.

## πŸ§ͺ Testing

- Unit tests port StreamChatAI's runner tests: a call runs once and only
for its person and install, an unsent result is sent again without
running the tool again, giving up after the attempts, finished calls,
the first tool of a name, and the registrations. Plus a call still
running not starting again, and a throwing tool reported as failed.
- On the iOS simulator and an Android emulator against a local
Vision-Agents router, "What time is it on my phone?":
- a plain question: the step waits for this install, the runner runs the
tool once and the router gets the result (0.2 s on iOS);
- with "Think harder": the step names no install and the socket answers
it.
- On the emulator the app once stopped getting chat updates mid-reply.
The channel had the whole reply, and reopening the conversation showed
it with the tool not run again. That's the sample's reconnect handling,
already planned for the polish pass.
- `yarn lint`, `yarn typecheck`, `yarn build` and the unit tests pass.

Router follow-up: `POST /v1/agents/sessions/{id}/responses` should take
`client_id` as `respond` does, and calypso's `responses.create` should
send it. Then the sample can drop the `respond` path.

## β˜‘οΈ Checklist

- [x] I have signed the [Stream
CLA](https://docs.google.com/forms/d/e/1FAIpQLScFKsKkAJI7mhCr7K9rEIOpqIDThrWxuvxnwUq2XkHyG154vQ/viewform)
(required)
- [ ] PR targets the `develop` branch (stacked on #3931)
- [ ] Documentation is updated
- [ ] New code is tested in main example apps, including all possible
scenarios
  - [ ] SampleApp iOS and Android
  - [x] Expo iOS and Android
isekovanic added a commit that referenced this pull request Nov 12, 2026
Ticket: https://linear.app/stream/issue/RN-485

## 🎯 Goal

Bring the AI agent work we shipped on `v9-release` to V10: the same
components, APIs and sample apps, adapted to stream-chat 10 and the V10
i18n.

The long-press Copy/Edit menu on the person's own messages isn't here.
It isn't on v9 either, and it comes later as a V10 feature.

## πŸ›  Implementation details

The branch is one commit per v9 PR, in merge order, then the V10-only
changes, so it can be reviewed commit by commit.

**Ported from v9**
- #3920 parse agent reply steps from message attachments
- #3921 AI translations through the chat SDK, sample on the agents
router
- #3924 agent reply steps: reasoning, tool calls, approvals
- #3926 native `AIAgent` module for the install id and tool-call sign-in
- #3928 message views, reply actions and icon overrides
- #3929 `AIConversation` on LegendList
- #3931 chat options, suggestions and attachment preview fixes
- #3933 client tool runner
- #3934 polish of agent conversations
- #3936 generated UI with an A2UI renderer
- #3944 follow-up polish of agent conversations
- #3958 A2UI list-row secrets and malformed messages
- #3957 conversation state with `AIConversationController`
- #3960 `AIConversation` in its own entry point
- #3963 the crew assistant demo app
- #3974 alignment with the other SDKs
- #3975 StrictMode-safe controller lifecycle and `custom.uiAction` docs
- #3976 crew drawer layout
- #3978 the samples on `@stream-io/stream-agents` 0.4.2, in place of
calypso and their own chat layer

Where `develop` had already changed the same lines (comment cleanups,
the i18n move, a11y docs), `develop`'s version wins.

**What's different on V10**
- **Fields under `custom`.** stream-chat 10 keeps `mime_type`,
`file_size`, `ai_generated`, `generating`, `ai_ui` and the step fields
under `custom`. `parseMessageParts` and `fromStreamMessage` lift them
with a small `withCustomFields` helper, as the React SDK does.
- **Translation keys.** The 55 AI and A2UI strings use catalog keys with
their English inline, named after React's
(`aiComponents.replyActions.copy.accessibilityLabel`,
`a2ui.modal.close.label`, …), and `keys.ts` is regenerated.
`useAITranslation` returns core's typed `t`.
- **Entry isolation.** The AI entry points may reach V10's translation
context. `@legendapp/list` and `react-native-keyboard-controller` are
AI-only packages.
- **LegendList stays at 3.7.1**, the version v9 resolves. 3.7.2 keeps
fewer rows mounted after a jump, so the reused-rows test now only
compares rows that stay mounted. It passes on both versions.
- **Sample data layer.** Both samples use stream-agents' `AgentChat`,
whose stream-chat 10 reader reads `channel.messagePaginator`, activates
the channel so the client reloads it after a reconnect, and loads older
pages with `toTail()`. The samples read the reply fields from `custom`,
and the crew sample uploads voice memos with `uploadChannelFile`.
- **Sample dependencies.** Both samples depend on
`@stream-io/state-store` and `use-sync-external-store`, and pin
`@stream-io/state-store` to one copy in `tsconfig` paths. They stay on
stream-chat `^10.0.1`: stream-agents asks for `^10.0.2`, but only when
it builds its own client, and the samples hand it theirs.

## 🎨 UI Changes

None compared with v9. The screenshots are in the v9 PRs above.

## πŸ§ͺ Testing

`yarn lint`, `yarn typecheck`, `test:typecheck`, the unit tests and
`yarn build` pass, and both samples typecheck.

On an Android emulator and an iOS simulator:
- **ChatGPT sample:** reasoning, the device time tool, the generated
Lisbon plan, the drawer, reopening a conversation from history, and
starters on a new one.
- **Crew sample:**
- all eight flows, including the approval cards, the checklist and form
actions, and the agent-first ones;
- the equipment deep link, photo troubleshooting and the team lead view;
  - voice memos, recorded on the simulator with `say` and answered;
  - the clock-out form on iOS.
- **Reconnect:** the network was dropped mid-reply on both samples. The
client recovered the active channel, and the full reply appeared.
- **Older messages:** with the page size briefly set to 3, older pages
loaded until there were none left.

After the move to stream-agents, both samples were checked again on both
platforms: replies, the device time tool, the agent-first Clock in flow,
the drawer, reopening from history, handbook sources, and an approval
card into A2UI. Reconnect, older pages and voice memos were tested on
the samples' previous chat layer, not again since.

Not tested here: voice mode (the simulator's speech recognition doesn't
hear `say`), dictation, Cal.com sign-in, and charts or photos in the
ChatGPT sample.

## β˜‘οΈ Checklist

- [x] I have signed the [Stream
CLA](https://docs.google.com/forms/d/e/1FAIpQLScFKsKkAJI7mhCr7K9rEIOpqIDThrWxuvxnwUq2XkHyG154vQ/viewform)
(required)
- [x] PR targets the `develop` branch
- [x] Documentation is updated
- [ ] New code is tested in main example apps, including all possible
scenarios
  - [ ] SampleApp iOS and Android
  - [x] Expo iOS and Android
@github-actions github-actions Bot mentioned this pull request Nov 12, 2026
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants