Skip to content

feat(ai-components): message views, reply actions and icon overrides - #3928

Merged
isekovanic merged 1 commit into
feat/ai-agent-nativefrom
feat/ai-message-views
Nov 6, 2026
Merged

isekovanic merged 1 commit into
feat/ai-agent-nativefrom
feat/ai-message-views

Conversation

@isekovanic

Copy link
Copy Markdown
Contributor

Ticket: https://linear.app/stream/issue/RN-485

Stacked on #3926.

🎯 Goal

Show the person's messages and the agent's replies the way the native AI SDKs do, with the actions under a finished reply. The APIs come from React's AI components (#3376). The look follows StreamChatAI and the Android AI SDK.

πŸ›  Implementation details

Views (props only, under message/)

  • AIUserMessage: the person's attachments, then their text in a bubble on the trailing side. With onEdit it shows an edit button.
  • AIAssistantMessage: a reply's steps, then the answer (typed out while it streams), then what the agent attached.
  • AIMessageAttachment:
    • An image is a 180 Γ— 180 thumbnail that doesn't open on tap, as on both native SDKs.
    • A file is a card with its type's icon (PDF, image, audio, video, archive or file), name, type and size. One at a web URL opens in the browser.
  • AIReplyActions: copy (shows a check for a moment), share through the system sheet, regenerate with onRegenerate, and a thumbs up/down rating with rating / onRate. Copy is hidden when the app has no clipboard module.

Customising

  • The AI theme has a new optional message section: styles, and colours under StreamChatAI's names (userMessageBackground, attachmentIconBackground, messageAction, …). The defaults come from the design tokens.
  • The theme also takes icons, which override any of the agent replies' 24 icons. This covers the step views from earlier PRs as well as the new ones. defaultAIIcons and useAIIcons are exported. mergeThemes carries icons through as they are, since the JSON copy it makes would drop them.
  • AIUserMessage and AIAssistantMessage take an Attachment component, like the attachment view factories on Swift and Android.

Also

  • Fixed: copying crashed with current expo-clipboard, which no longer has setString. The AI clipboard service now uses setStringAsync when it's there, as the core Expo package already does. This affected copying code blocks too.
  • New strings are in all 13 locales. As before, ar and he could use a native speaker's look.

Left for later

  • The native SDKs offer Copy and Edit on the person's message through a long-press menu (Swift's context menu, Android's dropdown). Until then, onEdit shows a button, as in React. If we want Swift's lifted preview, the menu needs a portal host of its own, which AIConversation can own in a later PR.

ExpoChatGPTSample

  • Renders the person's messages with AIUserMessage, and replies with AIAssistantMessage plus AIReplyActions once the router clears generating.
  • Ratings are kept for the session only, as in Juno: the router doesn't take ratings yet. Like Juno, the sample doesn't wire edit or regenerate.

🎨 UI Changes

The person's bubble, replies with their steps and action row, and image and file attachments.

πŸ§ͺ Testing

  • Unit tests cover:
    • the four views;
    • file size and details formatting;
    • the file type icons;
    • icon overrides, and icons surviving mergeThemes;
    • the Attachment slot.
  • On an Android emulator and the iOS simulator, against a local Vision-Agents router:
    • a web-search reply with its steps;
    • the actions appearing once the reply finished;
    • rating;
    • copy (pasting confirmed the text);
    • the share sheet;
    • image and PDF attachments;
    • the PDF opening in Chrome;
    • an image tap doing nothing.
  • The attachments were posted straight to the channel. Asking with a photo fails at the router ("personal conversations require a command ID") because of how the sample sends photos with calypso 0.2.3, not because of these views.
  • Not tested: edit and regenerate on a device (the sample doesn't use them), RTL and screen readers.
  • yarn lint, yarn typecheck, yarn build and yarn test:unit pass.

β˜‘οΈ Checklist

@isekovanic
isekovanic force-pushed the feat/ai-message-views branch from 7d7d9dd to 2789ab2 Compare November 5, 2026 16:57
@Stream-SDK-Bot

Copy link
Copy Markdown
Contributor

SDK Size

title develop branch diff status
js_bundle_size 2041 KB 2041 KB +213 B 🟒
ai_components_size 163 KB 226 KB +65237 B πŸ”΄

@isekovanic
isekovanic removed this pull request from stack #3942 November 6, 2026 14:16
@isekovanic
isekovanic added this pull request to stack #3949 November 6, 2026 14:16
@isekovanic
isekovanic removed this pull request from stack #3949 November 6, 2026 14:16
@isekovanic
isekovanic added this pull request to stack #3950 November 6, 2026 14:17
@isekovanic
isekovanic merged commit ff935fb into v9-release Nov 6, 2026
13 checks passed
@isekovanic
isekovanic deleted the feat/ai-message-views branch November 6, 2026 16:56
isekovanic added a commit that referenced this pull request Nov 6, 2026
Ticket: https://linear.app/stream/issue/RN-485

Stacked on #3928.

## 🎯 Goal

A conversation view for AI agents, laid out by turn like StreamChatAI's
`AIConversationView` and React's `AIConversation` (#3376): each question
starts a turn, the agent's reply writes out below it, and a new question
goes to the top of the screen without moving anything else.

## πŸ›  Implementation details

**`AIConversation`** (`conversation/`)
- Takes props only and never reads `<Channel>`, as in React: `messages`,
`pendingMessage`, `isThinking`, `loadOlderMessages`, `footer`,
`emptyContent`, `bottomInset`, `approver`, and the `UserMessage` /
`AssistantMessage` / `ReplyActions` slots plus `replyActions`.
- Split in three, like Swift's `AIConversationView` and its scroll view:
- `useConversationRows`: the data. Turns, a pending message until its
copy arrives, the thinking row, and a row cache so only changed messages
render again.
- `list/`: how the list behaves, as hooks that render nothing
(`usePinnedAnchor`, `useEndInView`, `useLoadOlder`, put together by
`useChatListController`), and an internal `AIChatList` that draws it.
Moving to another LegendList, such as keyboard-controller's, only
touches `AIChatList`.
  - `AIConversation` itself: the props, the slots and the row views.
- The list is Reanimated LegendList (`@legendapp/list` 3.7.1, JS only, a
regular dependency of the AI entry point; the isolation check keeps it
out of the main one), with recycling on.
- A reply's steps are keyed by message (new optional `id` on
`AIAssistantMessage`), since step ids like `r1` repeat across replies.
Reply actions are keyed too.
- The answer's typing starts afresh for a replaced text, so a reused row
shows its own text at once.
- The latest question is pinned with `anchoredEndSpace`, plus a content
floor (the question's position plus the list's tallest height at this
width). Without the floor, the list shrinking under the question (a
reasoning step folding, the thinking going, the keyboard closing) pulled
the question down before LegendList caught up. Thinking and the footer
are a row of their own for the same reason.
- Older messages load once the spinner has been in view for 300 ms, and
only when the list is at rest. On iOS, a page that lands while a finger
holds the list, or while it glides or bounces, pushes the view.
- Scroll-to-latest button (Swift's). It shows when the end of the latest
reply's content is out of view, not counting the space the pin leaves
under a short reply.

**Design tokens**
- Text in the AI components is sized from the chat SDK's tokens
(Md/Sm/Xs, 17/15/13 on iOS and 16/14/12 on Android), mapped to Swift's
font roles. Answers were RN's default 14 before. Headings scale from the
body as Swift's markdown does.
- The conversation's spacing comes from the tokens too.
- The isolation check now allows `theme/generated/light/StreamTokens*`,
which is plain data.

**Also**
- `fromStreamMessage` (React's) and `createFromStreamMessage`, which
returns the same `AIMessage` for the same Stream message object.
- `ArrowDown` icon, a `conversation` theme section (Swift's colours),
and two strings in all 13 locales. `ar` and `he` could use a native
speaker's look.
- Jest maps `@legendapp/list/{react-native,reanimated}` to their native
builds, since the exports map only has a `react-native` condition.

**ExpoChatGPTSample**
- Drops `<Channel>` and `<MessageList>` for `AIConversation`, as Juno
does. An `AgentChat` store watches the channel, follows message events,
pages older messages with `channel.query({ id_lt })` and keeps the live
reasoning.
- The question shows at once as the pending message, and "thinking"
comes from the session's state. Voice mode reads the session's state
instead of the AI indicator.

## 🎨 UI Changes

The conversation: turns, a new question pinned to the top with the reply
growing below, the spinner while older messages load, and the
scroll-to-latest button. Text is larger throughout (answers 17 on iOS,
16 on Android).

## πŸ§ͺ Testing

- Unit tests port React's `AIConversation` tests, and cover the rows
hook on its own, only changed rows re-rendering, the pending row keeping
its mount, loading older messages (delay, retry after a failure, waiting
for rest), repeated step ids across replies, a reused answer not typing
again, and `createFromStreamMessage`'s cache.
- On an Android emulator and the iOS simulator against a local
Vision-Agents router, and briefly on an iPhone:
- first and later questions, including asked with the keyboard up and
the keyboard then closed;
  - reasoning streaming and folding, and tool calls (web search);
  - the Linear sign-in card ("Not now"), stopping a reply, rating;
  - long replies and the scroll-to-latest button;
  - switching conversations;
  - loading older messages;
- recycling in a 34-turn conversation (rows reused 56 times): each row
showing its own message, no step state carried across replies, no answer
typing again.
- Not tested: TalkBack/VoiceOver and RTL; Connect on the sign-in card
(unchanged since #3926); approval cards and attachments in reused rows.
- Left for a polish pass: Swift's animations, the long-press Copy/Edit
menu, and trying react-native-keyboard-controller.
- `yarn lint`, `yarn typecheck`, `yarn build` and the unit tests pass.

## β˜‘οΈ Checklist

- [x] I have signed the [Stream
CLA](https://docs.google.com/forms/d/e/1FAIpQLScFKsKkAJI7mhCr7K9rEIOpqIDThrWxuvxnwUq2XkHyG154vQ/viewform)
(required)
- [ ] PR targets the `develop` branch (stacked on #3928)
- [ ] Documentation is updated
- [ ] New code is tested in main example apps, including all possible
scenarios
  - [ ] SampleApp iOS and Android
  - [x] Expo iOS and Android
isekovanic added a commit that referenced this pull request Nov 12, 2026
Ticket: https://linear.app/stream/issue/RN-485

## 🎯 Goal

Bring the AI agent work we shipped on `v9-release` to V10: the same
components, APIs and sample apps, adapted to stream-chat 10 and the V10
i18n.

The long-press Copy/Edit menu on the person's own messages isn't here.
It isn't on v9 either, and it comes later as a V10 feature.

## πŸ›  Implementation details

The branch is one commit per v9 PR, in merge order, then the V10-only
changes, so it can be reviewed commit by commit.

**Ported from v9**
- #3920 parse agent reply steps from message attachments
- #3921 AI translations through the chat SDK, sample on the agents
router
- #3924 agent reply steps: reasoning, tool calls, approvals
- #3926 native `AIAgent` module for the install id and tool-call sign-in
- #3928 message views, reply actions and icon overrides
- #3929 `AIConversation` on LegendList
- #3931 chat options, suggestions and attachment preview fixes
- #3933 client tool runner
- #3934 polish of agent conversations
- #3936 generated UI with an A2UI renderer
- #3944 follow-up polish of agent conversations
- #3958 A2UI list-row secrets and malformed messages
- #3957 conversation state with `AIConversationController`
- #3960 `AIConversation` in its own entry point
- #3963 the crew assistant demo app
- #3974 alignment with the other SDKs
- #3975 StrictMode-safe controller lifecycle and `custom.uiAction` docs
- #3976 crew drawer layout
- #3978 the samples on `@stream-io/stream-agents` 0.4.2, in place of
calypso and their own chat layer

Where `develop` had already changed the same lines (comment cleanups,
the i18n move, a11y docs), `develop`'s version wins.

**What's different on V10**
- **Fields under `custom`.** stream-chat 10 keeps `mime_type`,
`file_size`, `ai_generated`, `generating`, `ai_ui` and the step fields
under `custom`. `parseMessageParts` and `fromStreamMessage` lift them
with a small `withCustomFields` helper, as the React SDK does.
- **Translation keys.** The 55 AI and A2UI strings use catalog keys with
their English inline, named after React's
(`aiComponents.replyActions.copy.accessibilityLabel`,
`a2ui.modal.close.label`, …), and `keys.ts` is regenerated.
`useAITranslation` returns core's typed `t`.
- **Entry isolation.** The AI entry points may reach V10's translation
context. `@legendapp/list` and `react-native-keyboard-controller` are
AI-only packages.
- **LegendList stays at 3.7.1**, the version v9 resolves. 3.7.2 keeps
fewer rows mounted after a jump, so the reused-rows test now only
compares rows that stay mounted. It passes on both versions.
- **Sample data layer.** Both samples use stream-agents' `AgentChat`,
whose stream-chat 10 reader reads `channel.messagePaginator`, activates
the channel so the client reloads it after a reconnect, and loads older
pages with `toTail()`. The samples read the reply fields from `custom`,
and the crew sample uploads voice memos with `uploadChannelFile`.
- **Sample dependencies.** Both samples depend on
`@stream-io/state-store` and `use-sync-external-store`, and pin
`@stream-io/state-store` to one copy in `tsconfig` paths. They stay on
stream-chat `^10.0.1`: stream-agents asks for `^10.0.2`, but only when
it builds its own client, and the samples hand it theirs.

## 🎨 UI Changes

None compared with v9. The screenshots are in the v9 PRs above.

## πŸ§ͺ Testing

`yarn lint`, `yarn typecheck`, `test:typecheck`, the unit tests and
`yarn build` pass, and both samples typecheck.

On an Android emulator and an iOS simulator:
- **ChatGPT sample:** reasoning, the device time tool, the generated
Lisbon plan, the drawer, reopening a conversation from history, and
starters on a new one.
- **Crew sample:**
- all eight flows, including the approval cards, the checklist and form
actions, and the agent-first ones;
- the equipment deep link, photo troubleshooting and the team lead view;
  - voice memos, recorded on the simulator with `say` and answered;
  - the clock-out form on iOS.
- **Reconnect:** the network was dropped mid-reply on both samples. The
client recovered the active channel, and the full reply appeared.
- **Older messages:** with the page size briefly set to 3, older pages
loaded until there were none left.

After the move to stream-agents, both samples were checked again on both
platforms: replies, the device time tool, the agent-first Clock in flow,
the drawer, reopening from history, handbook sources, and an approval
card into A2UI. Reconnect, older pages and voice memos were tested on
the samples' previous chat layer, not again since.

Not tested here: voice mode (the simulator's speech recognition doesn't
hear `say`), dictation, Cal.com sign-in, and charts or photos in the
ChatGPT sample.

## β˜‘οΈ Checklist

- [x] I have signed the [Stream
CLA](https://docs.google.com/forms/d/e/1FAIpQLScFKsKkAJI7mhCr7K9rEIOpqIDThrWxuvxnwUq2XkHyG154vQ/viewform)
(required)
- [x] PR targets the `develop` branch
- [x] Documentation is updated
- [ ] New code is tested in main example apps, including all possible
scenarios
  - [ ] SampleApp iOS and Android
  - [x] Expo iOS and Android
@github-actions github-actions Bot mentioned this pull request Nov 12, 2026
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants