[Streams] Manually map new fields from schema editor #235919

tonyghiani · 2025-09-22T07:47:00Z

📓 Summary

This work adds a new user journey to allow Streams users to create new field mappings manually from the Schema editor.

Screen.Recording.2025-09-22.at.16.20.04.mov

Bonus point, the work introduces a change to the fields_metadata service to filter the retrieved fields by their source (ecs, otel, etc) to allow getting only the required lists.

🧪 Testing

Start Kibana with admin privileges
If Streams is disabled, enable the feature by toggling the Kibana Advanced Setting Streams
Ingest some logs data with node scripts/synthtrace.js sample_logs
Navigate to /app/streams and select one of the available streams (except the logs root stream, pick any other)
Click the Add field button on the top-right corner

elasticmachine · 2025-09-22T14:23:28Z

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

tonyghiani · 2025-09-22T14:29:23Z

...ared/streams_app/public/components/data_management/schema_editor/flyout/add_field_flyout.tsx

+  const { fieldsMetadata: rawFieldsMetadata } = useFieldsMetadata({
+    attributes: ['ignore_above', 'type', 'otel_equivalent'],
+    source: ['ecs', 'otel'],
+  });


Get all ecs/otel fields, selecting only required properties to reduce payload size and populate the suggestions selector.

tonyghiani · 2025-09-22T14:30:34Z

...red/streams_app/public/components/data_management/schema_editor/flyout/field_form_format.tsx

Minor design refactor to align with the ComboBox usage in the form and avoid multiple clicks between suggestions and custom options.

tonyghiani · 2025-09-22T14:31:54Z

.../plugins/shared/streams_app/public/components/data_management/schema_editor/flyout/index.tsx


  return (
-    <>
+    <EuiFlyout ownFocus onClose={onClose} aria-labelledby={flyoutId} maxWidth={500}>


Fixes focus issue detected on selectors.

…i/kibana into 212843-manually-map-new-fields

Kerry350

Couple of functionality questions, I'll look at the code next.

Is there any point in showing the fields that are going to be invalid because they already exist? Should we just filter them out?
When in an invalid state the "Add field" button doesn't do anything (which is correct), but it isn't explicitly disabled (so doesn't look disabled).

tonyghiani · 2025-09-23T11:55:41Z

Is there any point in showing the fields that are going to be invalid because they already exist? Should we just filter them out?

We could filter out the existing ones from the suggestions yeah, but users would still be able to manually write a colliding field name, so we should probably keep this around.

When in an invalid state the "Add field" button doesn't do anything (which is correct), but it isn't explicitly disabled (so doesn't look disabled).

It does trigger the validation every time it is clicked, but if the user updates the field affected by error, the error itself will disappear, and they'd still be able to click the add button. I'll look for an easy way to disable it when it's invalid and re-enable as soon as the field is touched/changed

Kerry350

Just a few functional questions / nits, but everything works as expected, nice work 👌

...orm/plugins/shared/fields_metadata/server/services/fields_metadata/fields_metadata_client.ts

...ared/streams_app/public/components/data_management/schema_editor/flyout/add_field_flyout.tsx

tonyghiani · 2025-09-23T12:30:31Z

@Kerry350 Thanks for the review. I addressed the minor changes and also improved the form behaviours on existing fields.

Kerry350 · 2025-09-23T13:08:22Z

Thanks for the review. I addressed the minor changes and also improved the form behaviours on existing fields.

Thanks, LGTM. Ship it 🚀

elasticmachine · 2025-09-23T14:48:30Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 585f6f5

Failed CI Steps

FTR Configs #129

Test Failures

[job] [logs] FTR Configs #129 / Space awareness package install "after all" hook in "package install"

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`streamsApp`	686	698	+12

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`fieldsMetadata`	47	48	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fieldsMetadata`	55.3KB	55.4KB	+31.0B
`streamsApp`	765.7KB	784.2KB	+18.5KB
total			+18.6KB

Unknown metric groups

API count

id	before	after	diff
`fieldsMetadata`	47	48	+1

History

💔 Build #342396 failed ef093d8
💔 Build #342127 failed a36f9f0

## 📓 Summary Closes elastic#212843 This work adds a new user journey to allow Streams users to create new field mappings manually from the Schema editor. https://github.com/user-attachments/assets/5a5ffb15-5085-48ab-84e2-4dcec87f7577 Bonus point, the work introduces a change to the fields_metadata service to filter the retrieved fields by their source (ecs, otel, etc) to allow getting only the required lists. ## 🧪 Testing - Start Kibana with admin privileges - If **Streams** is disabled, enable the feature by toggling the Kibana Advanced Setting `Streams` - Ingest some logs data with `node scripts/synthtrace.js sample_logs` - Navigate to `/app/streams` and select one of the available streams (except the `logs` root stream, pick any other) - Click the **Add field** button on the top-right corner

## 📓 Summary Closes #212843 This work adds a new user journey to allow Streams users to create new field mappings manually from the Schema editor. https://github.com/user-attachments/assets/5a5ffb15-5085-48ab-84e2-4dcec87f7577 Bonus point, the work introduces a change to the fields_metadata service to filter the retrieved fields by their source (ecs, otel, etc) to allow getting only the required lists. ## 🧪 Testing - Start Kibana with admin privileges - If **Streams** is disabled, enable the feature by toggling the Kibana Advanced Setting `Streams` - Ingest some logs data with `node scripts/synthtrace.js sample_logs` - Navigate to `/app/streams` and select one of the available streams (except the `logs` root stream, pick any other) - Click the **Add field** button on the top-right corner

## 📓 Summary Closes elastic#212843 This work adds a new user journey to allow Streams users to create new field mappings manually from the Schema editor. https://github.com/user-attachments/assets/5a5ffb15-5085-48ab-84e2-4dcec87f7577 Bonus point, the work introduces a change to the fields_metadata service to filter the retrieved fields by their source (ecs, otel, etc) to allow getting only the required lists. ## 🧪 Testing - Start Kibana with admin privileges - If **Streams** is disabled, enable the feature by toggling the Kibana Advanced Setting `Streams` - Ingest some logs data with `node scripts/synthtrace.js sample_logs` - Navigate to `/app/streams` and select one of the available streams (except the `logs` root stream, pick any other) - Click the **Add field** button on the top-right corner

tonyghiani added 4 commits September 19, 2025 15:19

feat: wip map new field flyout

adcba16

feat(fields_metadata): support source option for find operation

0085fd1

feat(schema_editor): mvp editor

f0479b6

fix(schema_editor): comment

baf51b8

tonyghiani added release_note:feature Makes this part of the condensed release notes Team:obs-onboarding Observability Onboarding Team Feature:Streams This is the label for the Streams Project labels Sep 22, 2025

tonyghiani added 4 commits September 22, 2025 12:36

feat(schema_editor): normalize field naming

0a090de

feat(schema_editor): keep ecs/otel fields for classic streams

bef40fd

test(schema_editor): add test for manually created field

6d17685

fix(schema_editor): super select focus

6fa05c5

tonyghiani marked this pull request as ready for review September 22, 2025 14:23

tonyghiani requested review from a team as code owners September 22, 2025 14:23

tonyghiani added the backport:skip This PR does not require backporting label Sep 22, 2025

Merge branch 'main' into 212843-manually-map-new-fields

5e83bf3

tonyghiani commented Sep 22, 2025

View reviewed changes

tonyghiani added 6 commits September 22, 2025 16:47

fix(schema_editor): linting and i18n

52b8d5c

Merge branch '212843-manually-map-new-fields' of github.com:tonyghian…

de843fb

…i/kibana into 212843-manually-map-new-fields

Merge branch 'main' into 212843-manually-map-new-fields

a36f9f0

fix(schema_editor): linting

5056fd4

Merge branch 'main' into 212843-manually-map-new-fields

ef093d8

fix(schema_editor): test

92e63d5

Kerry350 self-requested a review September 23, 2025 09:36

Kerry350 reviewed Sep 23, 2025

View reviewed changes

Kerry350 approved these changes Sep 23, 2025

View reviewed changes

...orm/plugins/shared/fields_metadata/server/services/fields_metadata/fields_metadata_client.ts Outdated Show resolved Hide resolved

...ared/streams_app/public/components/data_management/schema_editor/flyout/add_field_flyout.tsx Outdated Show resolved Hide resolved

tonyghiani added 2 commits September 23, 2025 14:25

refactor(schema_editor): address request changes

1a948f3

refactor(schema_editor): address request changes

585f6f5

tonyghiani merged commit d4f90ce into elastic:main Sep 23, 2025
12 checks passed

tonyghiani deleted the 212843-manually-map-new-fields branch September 23, 2025 14:52

kibanamachine added the v9.2.0 label Sep 23, 2025

[Streams] Manually map new fields from schema editor #235919

[Streams] Manually map new fields from schema editor #235919

Uh oh!

Conversation

tonyghiani commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📓 Summary

🧪 Testing

Uh oh!

elasticmachine commented Sep 22, 2025

Uh oh!

tonyghiani Sep 22, 2025

Choose a reason for hiding this comment

Uh oh!

tonyghiani Sep 22, 2025

Choose a reason for hiding this comment

Uh oh!

tonyghiani Sep 22, 2025

Choose a reason for hiding this comment

Uh oh!

Kerry350 left a comment

Choose a reason for hiding this comment

Uh oh!

tonyghiani commented Sep 23, 2025

Uh oh!

Kerry350 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tonyghiani commented Sep 23, 2025

Uh oh!

Kerry350 commented Sep 23, 2025

Uh oh!

elasticmachine commented Sep 23, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

API count

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tonyghiani commented Sep 22, 2025 •

edited

Loading