Update elastic-package to use Package Spec 3.0.0-rc1 and fix v3 packages #8115
Conversation
7a583d8 to
4cb9895
Compare
4cb9895 to
2cc3461
Compare
💔 Build Failed
Expand to view the summary
Build stats
Test stats 🧪
Steps errors
Expand to view the steps failures
|
🌐 Coverage report
|
This is not a lightweight switch either. Nested fields are indexed as separate documents, have their own query types (which are not well supported in Kibana in my own experience), and limit the field's use in visualizations. |
Yes, this is in my opinion a complicated case, because this field is modeled as an object, but it is also described as an array. Arrays of objects are not supported by Elasticsearch, so it can lead to unexpected results. One way to have an array of objects is to use the nested type, but as you said, it doesn't come without tradeoffs. I am going to try to relax this restriction, because it is causing many problems and probably the benefit of the correctness is not so big. |
|
@elastic/security-external-integrations one of the only pending issues is in the The issue is that this field is defined as Is this structure of arrays of arrays of objects needed or this could be flattened and then the current field mappings would work? |
If this structure is actually needed, I have prepared a PR to handle it: elastic/elastic-package#1498 |
|
Package hid_bravura_monitor - 1.15.1 containing this change is available at https://epr.elastic.co/search?package=hid_bravura_monitor |
|
Package juniper_srx - 1.18.1 containing this change is available at https://epr.elastic.co/search?package=juniper_srx |
|
Package netflow - 2.16.1 containing this change is available at https://epr.elastic.co/search?package=netflow |
|
Package network_traffic - 1.25.1 containing this change is available at https://epr.elastic.co/search?package=network_traffic |
|
Package osquery_manager - 1.10.1 containing this change is available at https://epr.elastic.co/search?package=osquery_manager |
|
Package panw_cortex_xdr - 1.21.1 containing this change is available at https://epr.elastic.co/search?package=panw_cortex_xdr |
|
Package suricata - 2.18.1 containing this change is available at https://epr.elastic.co/search?package=suricata |
|
Package sysmon_linux - 1.5.1 containing this change is available at https://epr.elastic.co/search?package=sysmon_linux |
|
Package zeek - 2.19.1 containing this change is available at https://epr.elastic.co/search?package=zeek |
| - name: tags.* | ||
| type: object | ||
| - name: tags | ||
| type: flattened |
There was a problem hiding this comment.
@jsoriano Is the mapping that is created by Fleet based on this change identical?
There was a problem hiding this comment.
No, it isn't. With this change we are providing an actual mapping to these fields.
Previous mapping was probably not the expected one, it was not generating any useful mapping:
"tags": {
"properties": {
"*": {
"type": "object"
}
}
}
Now it generates a flattened field:
"tags": {
"type": "flattened"
}
There was a problem hiding this comment.
I see, that previous one seems like a bug. Trying to remember what it exactly produced in Beats, my guess in this scenario was that just everything becomes a keyword. Is my assumption correct that the buggy mapping above did not have any affect? So if tags.foo came in, it was just a keyword? Now it will be flattened. One concern I have is that for some users this might be a breaking change as not all the same queries are supported: https://www.elastic.co/guide/en/elasticsearch/reference/current/flattened.html At the same time I like the idea of tags being of type flattened.
Main reason I comment to make sure we have thought through the potential implications of the change and in case users get hit by it, we have a plan for them ready. @tommyers-elastic
There was a problem hiding this comment.
I want to report a problem from a user due to changing the aws.dimension type from object to flattened:
they confirmed they can do search on the fields, but they cannot setup alerts on these flattened fields.
Proposed commit message
Fix field definitions in packages that were updated before v3 GA release, and are failing now with the first RC (see #8109, #8120)
Changes applied:
type: group. They don't generate any mapping, and they are detected as invalid objects withoutobject_type.Child mappings are imported too. Some examples of these object fields are
dns.answer, ornetwork.inner.object_typethat seem to fit as "tags" or "labels" use cases have been converted toflattened.object_typethat seem to fit as metrics have been converted totype: double.Packages with changes in fields include new changelog, so a new package will be released.
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Related issues
Screenshots