[Data Set Quality] Hide Data set details when dataStream is coming from remote cluster #220529

rStelmach · 2025-05-08T15:21:11Z

Because Data Set quality page does not currently work with CCS, logs coming from remote cluster have dataset quality link hidden in the flyout.

The fix would hide a Data Set Quality link for documents that are coming from remote clusters

Fixes #211602

…vigating-to-the-data-set-details-page-for-a-remote-data-stream

mohamedhamed-ahmed · 2025-05-20T23:23:34Z

...fied_doc_viewer/public/components/doc_viewer_logs_overview/logs_overview_degraded_fields.tsx

    prefix: qualityIssuesAccordionTitle,
  });

+  const isCCSRemoteIndex = isCCSRemoteIndexName(rawDoc._index ?? '');


QQ: Just curious if the _index can actually really be undefined? and do we want to display the Dataset Quality link in this case or not?

I can answer this from the Discover side, yes it can be for ES|QL mode if METADATA _index isn't used in the query. Although I'm not sure if the logs profile is currently showing the degraded fields section in ES|QL mode yet.

In DatasetQualityLink there is a line that would hide the link when dataStream is empty:
if (!locator || !dataStream) return null;
I think that if _index would be empty, the dataStream should be too ( correct me if I'm wrong)
If somehow dataStream would be present but _index wouldn't, the link itself would still work properly.

gbamparop · 2025-05-21T08:11:21Z

I've noticed some issues, but they are not related to this PR:

k8s.pod.ip and resource.attributes.k8s.pod.ip are displayed in the quality section as ignore fields, but only resource.attributes.k8s.pod.ip seems to be _ignored based on the doc.
The value of these fields in the data set quality section is empty, however, in the actual doc it's 0.0.0.0

You can check the doc in the release-oblt cluster.

Screen.Recording.2025-05-21.at.09.09.47.mov

gbamparop · 2025-05-21T08:18:03Z

k8s.pod.ip and resource.attributes.k8s.pod.ip are displayed in the quality section as ignore fields, but only resource.attributes.k8s.pod.ip seems to be _ignored based on the doc.

Actually they both appear as empty in the ignored_field_values section. @flash1293 do you know why this might be happening?

…vigating-to-the-data-set-details-page-for-a-remote-data-stream

kertal

While the code change LGTM, could you provide a bit more context, is this resolving an issue? it's labeled as bug, but this would be more suitable as a fix. And this should also be described in the PR description. Also a unit test would be great to cover it, thx 👍

rStelmach · 2025-05-29T10:23:44Z

@kertal

While the code change LGTM, could you provide a bit more context, is this resolving an issue? it's labeled as bug, but this would be more suitable as a fix. And this should also be described in the PR description. Also a unit test would be great to cover it, thx 👍

Hi, you are right. So the context is that now CCS clusters do not work with Dataset Quality (and for now there aren't any plans to change it). When you click that link it would take you to the page and it will result with an error. So for the logs that are coming from remote cluster we want to hide the link to dataset quality in the flyout.

I also added two tests to make sure it would work properly

…vigating-to-the-data-set-details-page-for-a-remote-data-stream

elasticmachine · 2025-06-04T08:17:51Z

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

AlexGPlay · 2025-06-04T13:00:51Z

.../shared/unified_doc_viewer/public/components/doc_viewer_logs_overview/logs_overview.test.tsx

+      const hitWithDataStream = buildHit({});
+
+      hitWithDataStream.raw.fields = sourceFields;


q: is it possible to do something like const hitWithDataStream = buildHit(sourceFields);?

AlexGPlay · 2025-06-04T13:02:21Z

.../shared/unified_doc_viewer/public/components/doc_viewer_logs_overview/logs_overview.test.tsx

+
+      const accordions = screen.getAllByTestId('unifiedDocViewLogsOverviewDegradedFieldsAccordion');
+      const accordion = accordions[accordions.length - 1];
+      const button = accordion.querySelector('button');


can we do within(accordion).getByRole('button')?

AlexGPlay · 2025-06-04T13:02:51Z

.../shared/unified_doc_viewer/public/components/doc_viewer_logs_overview/logs_overview.test.tsx

+      const accordion = accordions[accordions.length - 1];
+      const button = accordion.querySelector('button');
+
+      if (button) {


is the button not always there or is this just to appease the type checking?

AlexGPlay · 2025-06-04T13:03:25Z

.../shared/unified_doc_viewer/public/components/doc_viewer_logs_overview/logs_overview.test.tsx

+        act(() => {
+          button.click();
+        });


can we do something like

const user = userEvent.setup(); await user.click(button)

AlexGPlay · 2025-06-04T13:05:35Z

.../shared/unified_doc_viewer/public/components/doc_viewer_logs_overview/logs_overview.test.tsx

+      const remoteHit = buildHit({});
+
+      remoteHit.raw.fields = sourceFields;
+      remoteHit.raw._index = `remoteCluster:${DATA_STREAM_NAME}`;


same as before, is it possible to do buildHit(sourceFields) and maybe extend that mock factory to also take the index?

AlexGPlay

left some comments around testing - is it possible to somehow test this manually?

rStelmach · 2025-06-04T13:22:16Z

@AlexGPlay

Thank you for the review.
If you wanted to test it manually, you would have to :

Create through oblt a cluster and ingest some degraded or failed documents with synthrace (example command : node scripts/synthtrace degraded_logs.ts --target=https://elastic:M52oBciCkgNU9KJV9dnKiZAz@main-edge-oblt -ccs-r-wydbn.es.us-west2.gcp.elastic-cloud.com:443 --kibana=http://elastic:M52oBciCkgNU9KJV9dnKiZAz@localhost:5601)
Create another cluster with oblt ( best way is to do this through command in slack /create-ccs-cluster) and point that cluster to the one you have just created and ingested docs.
In the second cluster that is remotely connected create a dataview remote_cluster:* and pick a document like in the issue description video ( [Data Set Quality] 🐞 Error thrown when navigating to the Data Set details page for a remote data stream #211602 ) and in the flyout you should/shouldn't be able to see the Dataset Quality Link

Let me know if anything is unclear!

…vigating-to-the-data-set-details-page-for-a-remote-data-stream

AlexGPlay · 2025-06-05T06:46:01Z

.../shared/unified_doc_viewer/public/components/doc_viewer_logs_overview/logs_overview.test.tsx

+      const accordions = screen.getAllByTestId('unifiedDocViewLogsOverviewDegradedFieldsAccordion');
+      const accordion = accordions[accordions.length - 1];
      const button = accordion.querySelector('button');

-      if (button === null) {
-        return;
-      }
-      // Check the aria-expanded property of the button
-      const isExpanded = button.getAttribute('aria-expanded');
-      expect(isExpanded).toBe('false');
+      act(() => {
+        button?.click();
+      });


can we do the same we did for the other test here?

@AlexGPlay I updated all buttons in that test file so now it should be consistent

AlexGPlay · 2025-06-05T06:46:21Z

.../shared/unified_doc_viewer/public/components/doc_viewer_logs_overview/logs_overview.test.tsx

+      const accordions = screen.getAllByTestId('unifiedDocViewLogsOverviewDegradedFieldsAccordion');
+      const accordion = accordions[accordions.length - 1];
+      const button = accordion.querySelector('button');
+
+      act(() => {
+        button?.click();
+      });


same here please 🙏

AlexGPlay · 2025-06-06T08:34:08Z

.../shared/unified_doc_viewer/public/components/doc_viewer_logs_overview/logs_overview.test.tsx

+      act(() => {
+        button?.click();
+      });


Suggested change

act(() => {

button?.click();

});

const user = userEvent.setup();

await user.click(button);

Oh you meant those changes 😄
Got it now

yeah, no worries, other than this i think it looks good

AlexGPlay · 2025-06-06T08:34:20Z

.../shared/unified_doc_viewer/public/components/doc_viewer_logs_overview/logs_overview.test.tsx

+      act(() => {
+        button?.click();
+      });


Suggested change

act(() => {

button?.click();

});

const user = userEvent.setup();

await user.click(button);

Right now it should be ok

AlexGPlay

thank you for taking your time getting the tests to a better state 😄

…vigating-to-the-data-set-details-page-for-a-remote-data-stream

elasticmachine · 2025-06-09T13:51:53Z

💚 Build Succeeded

Buildkite Build
Commit: 48253f0

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`unifiedDocViewer`	220.2KB	220.3KB	+95.0B

History

💛 Build #306207 was flaky dfe8058
💔 Build #306140 failed 3ca7582
💔 Build #306080 failed eb502de
💛 Build #305292 was flaky 08d66b4
💚 Build #304195 succeeded 338bce0
💚 Build #303298 succeeded 8a41dc6

yngrdyn · 2025-06-10T12:22:45Z

/oblt-deploy

yngrdyn

LGTM, I tested the changes and the link is hidden when we are checking a log coming from a remote cluster and is shown when the log is local

Screen.Recording.2025-06-11.at.11.18.00.mov

kibanamachine · 2025-06-11T09:20:07Z

Starting backport for target branches: 8.19

https://github.com/elastic/kibana/actions/runs/15581035330

…om remote cluster (elastic#220529) Because Data Set quality page does not currently work with CCS, logs coming from remote cluster have dataset quality link hidden in the flyout. The fix would hide a Data Set Quality link for documents that are coming from remote clusters Fixes elastic#211602 --------- Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit fa400a4)

kibanamachine · 2025-06-11T09:26:59Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…ming from remote cluster (#220529) (#223376) # Backport This will backport the following commits from `main` to `8.19`: - [[Data Set Quality] Hide Data set details when dataStream is coming from remote cluster (#220529)](#220529)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Robert Stelmach <[email protected]>

…om remote cluster (elastic#220529) Because Data Set quality page does not currently work with CCS, logs coming from remote cluster have dataset quality link hidden in the flyout. The fix would hide a Data Set Quality link for documents that are coming from remote clusters Fixes elastic#211602 --------- Co-authored-by: kibanamachine <[email protected]>

rStelmach added 2 commits May 8, 2025 17:20

Hide Data set details when dataStream is coming from remote cluster

028684a

Add proper function to detect if dataStream comes from remote cluster

38585aa

rStelmach marked this pull request as ready for review May 20, 2025 11:20

rStelmach requested a review from a team as a code owner May 20, 2025 11:20

Merge branch 'main' into 211602-data-set-quality-error-thrown-when-na…

32b5d69

…vigating-to-the-data-set-details-page-for-a-remote-data-stream

rStelmach added release_note:fix backport:version Backport to applied version labels v9.1.0 v8.19.0 labels May 20, 2025

rStelmach linked an issue May 20, 2025 that may be closed by this pull request

[Data Set Quality] 🐞 Error thrown when navigating to the Data Set details page for a remote data stream #211602

Closed

rStelmach added bug Fixes for quality problems that affect the customer experience Feature:Dataset Health labels May 20, 2025

rStelmach changed the title ~~Hide Data set details when dataStream is coming from remote cluster~~ [Data Set Quality] Hide Data set details when dataStream is coming from remote cluster May 20, 2025

[CI] Auto-commit changed files from 'node scripts/notice'

0347195

elastic-vault-github-plugin-prod bot requested a review from a team as a code owner May 20, 2025 11:37

mohamedhamed-ahmed reviewed May 20, 2025

View reviewed changes

Merge branch 'main' into 211602-data-set-quality-error-thrown-when-na…

6e10172

…vigating-to-the-data-set-details-page-for-a-remote-data-stream

kertal reviewed May 28, 2025

View reviewed changes

add test to check if hiding datasetquality link works

8a41dc6

rStelmach added 2 commits May 29, 2025 15:34

Merge branch 'main' into 211602-data-set-quality-error-thrown-when-na…

6e327eb

…vigating-to-the-data-set-details-page-for-a-remote-data-stream

Merge branch 'main' into 211602-data-set-quality-error-thrown-when-na…

338bce0

…vigating-to-the-data-set-details-page-for-a-remote-data-stream

kertal requested a review from a team June 3, 2025 14:34

rStelmach added the Team:obs-onboarding Observability Onboarding Team label Jun 4, 2025

rStelmach removed the bug Fixes for quality problems that affect the customer experience label Jun 4, 2025

AlexGPlay reviewed Jun 4, 2025

View reviewed changes

rStelmach and others added 2 commits June 4, 2025 16:56

adjust tests

21a6177

Merge branch 'main' into 211602-data-set-quality-error-thrown-when-na…

08d66b4

…vigating-to-the-data-set-details-page-for-a-remote-data-stream

AlexGPlay reviewed Jun 5, 2025

View reviewed changes

update button in tests

712b415

AlexGPlay reviewed Jun 6, 2025

View reviewed changes

adjust test buttons

eb502de

AlexGPlay approved these changes Jun 6, 2025

View reviewed changes

rStelmach added 3 commits June 6, 2025 12:26

Merge branch 'main' into 211602-data-set-quality-error-thrown-when-na…

3ca7582

…vigating-to-the-data-set-details-page-for-a-remote-data-stream

Merge branch 'main' into 211602-data-set-quality-error-thrown-when-na…

dfe8058

…vigating-to-the-data-set-details-page-for-a-remote-data-stream

Merge branch 'main' into 211602-data-set-quality-error-thrown-when-na…

48253f0

…vigating-to-the-data-set-details-page-for-a-remote-data-stream

yngrdyn approved these changes Jun 11, 2025

View reviewed changes

rStelmach merged commit fa400a4 into elastic:main Jun 11, 2025
10 checks passed

kibanamachine mentioned this pull request Jun 11, 2025

[8.19] [Data Set Quality] Hide Data set details when dataStream is coming from remote cluster (#220529) #223376

Merged

		const hitWithDataStream = buildHit({});

		hitWithDataStream.raw.fields = sourceFields;

[Data Set Quality] Hide Data set details when dataStream is coming from remote cluster #220529

[Data Set Quality] Hide Data set details when dataStream is coming from remote cluster #220529

Uh oh!

Conversation

rStelmach commented May 8, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gbamparop commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gbamparop commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kertal left a comment

Choose a reason for hiding this comment

Uh oh!

rStelmach commented May 29, 2025

Uh oh!

elasticmachine commented Jun 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AlexGPlay left a comment

Choose a reason for hiding this comment

Uh oh!

rStelmach commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AlexGPlay left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jun 9, 2025

💚 Build Succeeded

Metrics [docs]

Async chunks

History

Uh oh!

yngrdyn commented Jun 10, 2025

Uh oh!

yngrdyn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kibanamachine commented Jun 11, 2025

Uh oh!

kibanamachine commented Jun 11, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

rStelmach commented May 8, 2025 •

edited by kibanamachine

Loading

gbamparop commented May 21, 2025 •

edited

Loading

gbamparop commented May 21, 2025 •

edited

Loading

rStelmach commented Jun 4, 2025 •

edited

Loading