Skip to content

[Fleet] Expand subfields of nested objects when generating template #191730

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jsoriano merged 6 commits into from
Aug 30, 2024
Merged

[Fleet] Expand subfields of nested objects when generating template #191730

jsoriano merged 6 commits into from
Aug 30, 2024

Conversation

@jsoriano
Copy link
Member

@jsoriano jsoriano commented Aug 29, 2024 •
edited by kibanamachine
Loading

Summary

There are definitions of nested objects whose fields are defined as subfields, like this:

  - name: a
    type: nested
    fields:
    - name: b
      type: keyword

This should generate a template with the subfields as subproperties:

      "properties": {
        ...
        "a": {
          "type": "nested",
          "properties": {
            "b": {
              "ignore_above": 1024,
              "type": "keyword",
            },
          },
        },
      },

This change adds support for this. Without it the nested object is empty, without subfields, what is unexpected.

See elastic/package-spec#784 for more context.

Checklist

Delete any items that are not applicable to this PR.

Risk Matrix

Risk Probability Severity Mitigation/Notes
This change will add the expected mappings to nested subfields that currently don't have any mapping. This could cause mapping conflicts if any package is producing values with the unexpected type. Low Medium If this happens this will be detected by the next integrations daily build with the latest snapshot.

For maintainers

@jsoriano jsoriano added bug Fixes for quality problems that affect the customer experience Team:Fleet Team label for Observability Data Collection Fleet team backport:prev-minor labels Aug 29, 2024
@jsoriano jsoriano self-assigned this Aug 29, 2024
@jsoriano jsoriano requested a review from a team as a code owner August 29, 2024 12:15
@elasticmachine
Copy link
Contributor

elasticmachine commented Aug 29, 2024

Pinging @elastic/fleet (Team:Fleet)

@obltmachine
Copy link

obltmachine commented Aug 29, 2024

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@jsoriano jsoriano force-pushed the nested-group-nested branch from 80ab845 to f5a1a65 Compare August 29, 2024 12:16
@jsoriano jsoriano mentioned this pull request Aug 29, 2024
Closed
juliaElastic
juliaElastic approved these changes Aug 29, 2024
View reviewed changes
Copy link
Contributor

@juliaElastic juliaElastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, should we add tests on group-nested too?

@jsoriano
Copy link
Member Author

jsoriano commented Aug 29, 2024

should we add tests on group-nested too?

group-nested seems to be only used internally, this is not allowed in packages configuration.

@jsoriano jsoriano enabled auto-merge (squash) August 29, 2024 22:14
@jsoriano jsoriano force-pushed the nested-group-nested branch from 43c8e96 to e5ab20d Compare August 30, 2024 17:13
@jsoriano jsoriano requested a review from juliaElastic August 30, 2024 17:18
@kibana-ci
Copy link

kibana-ci commented Aug 30, 2024

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #43 / Serverless security API cloud_security_posture GET internal/cloud_security_posture/rules/_find Should return 500 error code when provide both package policy id and benchmark id

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jsoriano

@jsoriano jsoriano merged commit cdb1eb8 into elastic:main Aug 30, 2024
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Aug 30, 2024
@jsoriano
[Fleet] Expand subfields of nested objects when generating template (e…
cc0f3eb 
…lastic#191730)

There are definitions of nested objects whose fields are defined as
subfields, like this:
```
  - name: a
    type: nested
    fields:
    - name: b
      type: keyword
```
This should generate a template with the subfields as subproperties:
```
      "properties": {
        ...
        "a": {
          "type": "nested",
          "properties": {
            "b": {
              "ignore_above": 1024,
              "type": "keyword",
            },
          },
        },
      },
```
This change adds support for this. Without it the nested object is
empty, without subfields, what is unexpected.

See elastic/package-spec#784 for more context.

(cherry picked from commit cdb1eb8)
@kibanamachine kibanamachine mentioned this pull request Aug 30, 2024
Merged
@kibanamachine
Copy link
Contributor

kibanamachine commented Aug 30, 2024

💚 All backports created successfully

Status Branch Result
✅ 8.15

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Aug 30, 2024
@kibanamachine @jsoriano
[8.15] [Fleet] Expand subfields of nested objects when generating tem…
ab6bb9c 
…plate (#191730) (#191879)

# Backport

This will backport the following commits from `main` to `8.15`:
- [[Fleet] Expand subfields of nested objects when generating template
(#191730)](#191730)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Jaime Soriano
Pastor","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-08-30T19:25:15Z","message":"[Fleet]
Expand subfields of nested objects when generating template
(#191730)\n\nThere are definitions of nested objects whose fields are
defined as\r\nsubfields, like this:\r\n```\r\n - name: a\r\n type:
nested\r\n fields:\r\n - name: b\r\n type: keyword\r\n```\r\nThis should
generate a template with the subfields as subproperties:\r\n```\r\n
\"properties\": {\r\n ...\r\n \"a\": {\r\n \"type\": \"nested\",\r\n
\"properties\": {\r\n \"b\": {\r\n \"ignore_above\": 1024,\r\n \"type\":
\"keyword\",\r\n },\r\n },\r\n },\r\n },\r\n```\r\nThis change adds
support for this. Without it the nested object is\r\nempty, without
subfields, what is unexpected.\r\n\r\nSee
elastic/package-spec#784 for more
context.","sha":"cdb1eb8fe475c3f1dad0805586c06775538d2d40","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:fix","Team:Fleet","backport:prev-minor","v8.16.0"],"title":"[Fleet]
Expand subfields of nested objects when generating
template","number":191730,"url":"https://github.com/elastic/kibana/pull/191730","mergeCommit":{"message":"[Fleet]
Expand subfields of nested objects when generating template
(#191730)\n\nThere are definitions of nested objects whose fields are
defined as\r\nsubfields, like this:\r\n```\r\n - name: a\r\n type:
nested\r\n fields:\r\n - name: b\r\n type: keyword\r\n```\r\nThis should
generate a template with the subfields as subproperties:\r\n```\r\n
\"properties\": {\r\n ...\r\n \"a\": {\r\n \"type\": \"nested\",\r\n
\"properties\": {\r\n \"b\": {\r\n \"ignore_above\": 1024,\r\n \"type\":
\"keyword\",\r\n },\r\n },\r\n },\r\n },\r\n```\r\nThis change adds
support for this. Without it the nested object is\r\nempty, without
subfields, what is unexpected.\r\n\r\nSee
elastic/package-spec#784 for more
context.","sha":"cdb1eb8fe475c3f1dad0805586c06775538d2d40"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/191730","number":191730,"mergeCommit":{"message":"[Fleet]
Expand subfields of nested objects when generating template
(#191730)\n\nThere are definitions of nested objects whose fields are
defined as\r\nsubfields, like this:\r\n```\r\n - name: a\r\n type:
nested\r\n fields:\r\n - name: b\r\n type: keyword\r\n```\r\nThis should
generate a template with the subfields as subproperties:\r\n```\r\n
\"properties\": {\r\n ...\r\n \"a\": {\r\n \"type\": \"nested\",\r\n
\"properties\": {\r\n \"b\": {\r\n \"ignore_above\": 1024,\r\n \"type\":
\"keyword\",\r\n },\r\n },\r\n },\r\n },\r\n```\r\nThis change adds
support for this. Without it the nested object is\r\nempty, without
subfields, what is unexpected.\r\n\r\nSee
elastic/package-spec#784 for more
context.","sha":"cdb1eb8fe475c3f1dad0805586c06775538d2d40"}}]}]
BACKPORT-->

Co-authored-by: Jaime Soriano Pastor <[email protected]>
jsoriano added a commit to jsoriano/kibana that referenced this pull request Sep 2, 2024
@jsoriano
Revert "[Fleet] Expand subfields of nested objects when generating te…
f31ffec 
…mplate (elastic#191730)"

This reverts commit cdb1eb8.
jsoriano added a commit that referenced this pull request Sep 2, 2024
@jsoriano
Revert "[Fleet] Expand subfields of nested objects when generating te…
7cf77b3 
…mplate (#191730)" (#191897)

This reverts commit cdb1eb8.

It is a bugfix but was not intended to be released in a patch version.
@jsoriano jsoriano mentioned this pull request Sep 5, 2024
Merged
4 tasks
jsoriano added a commit to jsoriano/ingest-docs that referenced this pull request Sep 5, 2024
@jsoriano
Add release notes for elastic/kibana#191730
11c68e5
@jsoriano jsoriano mentioned this pull request Sep 5, 2024
Merged
jsoriano added a commit to elastic/ingest-docs that referenced this pull request Sep 5, 2024
@jsoriano
Add release notes for elastic/kibana#191730 (#1292)
4928797
mergify bot pushed a commit to elastic/ingest-docs that referenced this pull request Sep 5, 2024
@jsoriano @mergify
Add release notes for elastic/kibana#191730 (#1292)
2d0b4ed 
(cherry picked from commit 4928797)
@mergify mergify bot mentioned this pull request Sep 5, 2024
Merged
jsoriano added a commit to elastic/integrations that referenced this pull request Sep 6, 2024
@jsoriano
Adapt definitions of nested subfields to current Fleet implementation (…
0751dc1 
…#11016)

These mappings were correctly defined as expected by the spec, but Fleet was only
installing empty nested objects. To workaround that, subfields can be moved to have
their own definitions.

Issue in Fleet is fixed in elastic/kibana#191730, we can apply this workaround for
current versions of the stack.
jsoriano added a commit to jsoriano/kibana that referenced this pull request Sep 6, 2024
@jsoriano
Reapply "[Fleet] Expand subfields of nested objects when generating t…
2f78739 
…emplate (elastic#191730)" (elastic#191897)

This reverts commit 7cf77b3.
jsoriano added a commit that referenced this pull request Sep 6, 2024
@jsoriano
Reapply "[Fleet] Expand subfields of nested objects when generating t…
fdbaff1 
…emplate (#191730)" (#191897) (#192246)

This change was released at the end in 8.15.1, so let's keep it the
branch.
    
There are definitions of nested objects whose fields are defined as
subfields, like this:
```
  - name: a
    type: nested
    fields:
    - name: b
      type: keyword
```
This should generate a template with the subfields as subproperties:
```
      "properties": {
        ...
        "a": {
          "type": "nested",
          "properties": {
            "b": {
              "ignore_above": 1024,
              "type": "keyword",
            },
          },
        },
      },
```
This change adds support for this. Without it the nested object is
empty, without subfields, what is unexpected.

See elastic/package-spec#784 for more context.

This change was originally reverted in 8.15 in  #191897

Release notes were manually added in elastic/ingest-docs#1292
@jsoriano jsoriano mentioned this pull request Sep 6, 2024
Merged
jsoriano added a commit to elastic/integrations that referenced this pull request Sep 9, 2024
@jsoriano
[proofpoint_on_demand] Adapt definitions of nested subfields (#11031)
92f726e 
Adapt definitions of subfields of nested objects so they are actually installed by
Fleet. New mappings are otherwise equivalent.

These mappings were correctly defined as expected by the spec, but Fleet was
only installing empty nested objects. To workaround that, subfields can be moved
to have their own definitions.

Issue in Fleet is fixed in elastic/kibana#191730, but we can apply this workaround
for older versions of the stack.
jsoriano added a commit to elastic/ingest-docs that referenced this pull request Sep 9, 2024
@mergify @jsoriano
Add release notes for elastic/kibana#191730 (#1292) (#1293)
a9c3be5 
(cherry picked from commit 4928797)

Co-authored-by: Jaime Soriano Pastor <[email protected]>
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
@jsoriano
Adapt definitions of nested subfields to current Fleet implementation (…
998c7de 
…elastic#11016)

These mappings were correctly defined as expected by the spec, but Fleet was only
installing empty nested objects. To workaround that, subfields can be moved to have
their own definitions.

Issue in Fleet is fixed in elastic/kibana#191730, we can apply this workaround for
current versions of the stack.
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
@jsoriano
[proofpoint_on_demand] Adapt definitions of nested subfields (elastic…
c5c5f82 
…#11031)

Adapt definitions of subfields of nested objects so they are actually installed by
Fleet. New mappings are otherwise equivalent.

These mappings were correctly defined as expected by the spec, but Fleet was
only installing empty nested objects. To workaround that, subfields can be moved
to have their own definitions.

Issue in Fleet is fixed in elastic/kibana#191730, but we can apply this workaround
for older versions of the stack.
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
@jsoriano
Adapt definitions of nested subfields to current Fleet implementation (…
62453d5 
…elastic#11016)

These mappings were correctly defined as expected by the spec, but Fleet was only
installing empty nested objects. To workaround that, subfields can be moved to have
their own definitions.

Issue in Fleet is fixed in elastic/kibana#191730, we can apply this workaround for
current versions of the stack.
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
@jsoriano
[proofpoint_on_demand] Adapt definitions of nested subfields (elastic…
6bf5a4d 
…#11031)

Adapt definitions of subfields of nested objects so they are actually installed by
Fleet. New mappings are otherwise equivalent.

These mappings were correctly defined as expected by the spec, but Fleet was
only installing empty nested objects. To workaround that, subfields can be moved
to have their own definitions.

Issue in Fleet is fixed in elastic/kibana#191730, but we can apply this workaround
for older versions of the stack.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@juliaElastic juliaElastic Awaiting requested review from juliaElastic

Assignees

@jsoriano jsoriano

Labels

bug Fixes for quality problems that affect the customer experience release_note:fix Team:Fleet Team label for Observability Data Collection Fleet team v8.15.1 v8.16.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@jsoriano @elasticmachine @obltmachine @kibana-ci @kibanamachine @juliaElastic