Fix broken EDOT JVM metrics dashboard when classic agent metrics are present #224052
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services) |
jennypavlova
approved these changes
Jun 16, 2025
Member
jennypavlova
left a comment
There was a problem hiding this comment.
LGTM, just a question ⬇️
...ins/apm/public/components/app/metrics/static_dashboard/dashboards/otel_native-edot-java.json
Outdated
Show resolved
Hide resolved
Member
|
Note: I added the same backport/release notes labels as in #223539 :) |
Contributor
Author
|
Got the confirmation from @felixbarny that the |
Contributor
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]Async chunks
History
|
Contributor
|
Starting backport for target branches: 8.18, 8.19, 9.0 |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jun 19, 2025
…present (elastic#224052) Co-authored-by: jennypavlova <[email protected]> (cherry picked from commit 7818df9)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jun 19, 2025
…present (elastic#224052) Co-authored-by: jennypavlova <[email protected]> (cherry picked from commit 7818df9)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jun 19, 2025
…present (elastic#224052) Co-authored-by: jennypavlova <[email protected]> (cherry picked from commit 7818df9)
Contributor
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Jun 19, 2025
…cs are present (#224052) (#224527) # Backport This will backport the following commits from `main` to `8.18`: - [Fix broken EDOT JVM metrics dashboard when classic agent metrics are present (#224052)](#224052) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Jonas Kunz","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-06-19T08:56:56Z","message":"Fix broken EDOT JVM metrics dashboard when classic agent metrics are present (#224052)\n\nCo-authored-by: jennypavlova <[email protected]>","sha":"7818df99305093219006b6d1f8c4b375e0b5a07f","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:obs-ux-infra_services","backport:version","v9.1.0","v8.19.0","v9.0.3","v8.18.3"],"title":"Fix broken EDOT JVM metrics dashboard when classic agent metrics are present","number":224052,"url":"https://github.com/elastic/kibana/pull/224052","mergeCommit":{"message":"Fix broken EDOT JVM metrics dashboard when classic agent metrics are present (#224052)\n\nCo-authored-by: jennypavlova <[email protected]>","sha":"7818df99305093219006b6d1f8c4b375e0b5a07f"}},"sourceBranch":"main","suggestedTargetBranches":["8.19","9.0","8.18"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/224052","number":224052,"mergeCommit":{"message":"Fix broken EDOT JVM metrics dashboard when classic agent metrics are present (#224052)\n\nCo-authored-by: jennypavlova <[email protected]>","sha":"7818df99305093219006b6d1f8c4b375e0b5a07f"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.0","label":"v9.0.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Jonas Kunz <[email protected]> Co-authored-by: jennypavlova <[email protected]>
kibanamachine
added a commit
that referenced
this pull request
Jun 19, 2025
…s are present (#224052) (#224529) # Backport This will backport the following commits from `main` to `9.0`: - [Fix broken EDOT JVM metrics dashboard when classic agent metrics are present (#224052)](#224052) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Jonas Kunz","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-06-19T08:56:56Z","message":"Fix broken EDOT JVM metrics dashboard when classic agent metrics are present (#224052)\n\nCo-authored-by: jennypavlova <[email protected]>","sha":"7818df99305093219006b6d1f8c4b375e0b5a07f","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:obs-ux-infra_services","backport:version","v9.1.0","v8.19.0","v9.0.3","v8.18.3"],"title":"Fix broken EDOT JVM metrics dashboard when classic agent metrics are present","number":224052,"url":"https://github.com/elastic/kibana/pull/224052","mergeCommit":{"message":"Fix broken EDOT JVM metrics dashboard when classic agent metrics are present (#224052)\n\nCo-authored-by: jennypavlova <[email protected]>","sha":"7818df99305093219006b6d1f8c4b375e0b5a07f"}},"sourceBranch":"main","suggestedTargetBranches":["8.19","9.0","8.18"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/224052","number":224052,"mergeCommit":{"message":"Fix broken EDOT JVM metrics dashboard when classic agent metrics are present (#224052)\n\nCo-authored-by: jennypavlova <[email protected]>","sha":"7818df99305093219006b6d1f8c4b375e0b5a07f"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.0","label":"v9.0.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Jonas Kunz <[email protected]> Co-authored-by: jennypavlova <[email protected]>
kibanamachine
added a commit
that referenced
this pull request
Jun 19, 2025
…cs are present (#224052) (#224528) # Backport This will backport the following commits from `main` to `8.19`: - [Fix broken EDOT JVM metrics dashboard when classic agent metrics are present (#224052)](#224052) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Jonas Kunz","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-06-19T08:56:56Z","message":"Fix broken EDOT JVM metrics dashboard when classic agent metrics are present (#224052)\n\nCo-authored-by: jennypavlova <[email protected]>","sha":"7818df99305093219006b6d1f8c4b375e0b5a07f","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:obs-ux-infra_services","backport:version","v9.1.0","v8.19.0","v9.0.3","v8.18.3"],"title":"Fix broken EDOT JVM metrics dashboard when classic agent metrics are present","number":224052,"url":"https://github.com/elastic/kibana/pull/224052","mergeCommit":{"message":"Fix broken EDOT JVM metrics dashboard when classic agent metrics are present (#224052)\n\nCo-authored-by: jennypavlova <[email protected]>","sha":"7818df99305093219006b6d1f8c4b375e0b5a07f"}},"sourceBranch":"main","suggestedTargetBranches":["8.19","9.0","8.18"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/224052","number":224052,"mergeCommit":{"message":"Fix broken EDOT JVM metrics dashboard when classic agent metrics are present (#224052)\n\nCo-authored-by: jennypavlova <[email protected]>","sha":"7818df99305093219006b6d1f8c4b375e0b5a07f"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.0","label":"v9.0.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Jonas Kunz <[email protected]> Co-authored-by: jennypavlova <[email protected]> Co-authored-by: Sergi Romeu <[email protected]>
Member
|
This PR didn't make it on time for the latest v8.18.3 and v9.0.3 BC. Updating the labels |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
A customer reported that the EDOT JVM metrics tab is broken when his cluster contains both metrics from the classic APM java agent and opentelemetry java agent metrics ingested via the EDOT collector.
Some panels in the dashboard would show the following error:
We figured out that this seems to happen due to a metric name clash between the JVM metrics reported by the classic APM agent and the OpenTelemetry Java agent: Both report a
jvm.thread.countmetric, but the classic agent reports it asdouble, while the opentelemetry one useslong.In the EDOT JVM metrics dashboard we use the very wide
metrics-*index pattern in the ES|QL queries and later on filter down the data in the query. For the panels usingjvm.thread.countthis causes ES|QL to rightly complain about the different types for the values found for that column, because we by accident include the classic APM data stream indices.This PR fixes this behaviour by switching to the more narrow
metrics-generic.otel-*metrics-*.otel-*index pattern.I've manually reproduced the issue and tried the fix using the following setup:
This setup (without this PR) reproduces the issue, but with a slightly different error message. I assume this is due to some changes in the ES|QL implementation in the snapshot, but the root cause is the same:
With the changes from this PR, the dashboard renders correctly.
This change overall is similar to #223539.
Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n supportDocumentation was added for features that require explanation or tutorialsUnit or functional tests were updated or added to match the most common scenariosIf a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker listThis was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. Therelease_note:breakinglabel should be applied in these situations.Flaky Test Runner was used on any tests changedrelease_note:*label is applied per the guidelinesIdentify risks
We need to make sure that the
metrics-generic.otel-*metrics-*.otel-*index pattern is actually the correct one to use and is stable to use.I'd ask @elastic/obs-ds-intake-services to confirm this.