Misleading implementation of mappings of nested object subfields

[jsoriano]

We have found that fields with type: nested and subfields don't get the mappings for the subfields in the final index template, this can lead to unexpected mappings.

That is that for example for a mapping with several subfields, this empty nested mapping is generated:

"whats": {
    "type": "nested"
}

In combination with dynamic mappings introduced by ecs@mappings this can lead subfields to have completely unexpected mappings.

While investigating we have found that the implementation in Fleet has two different codepaths for type: nested and for type: group-nested. For nested it ignores any subfieds, for group-nested it generates the expected mappings. We have also found that package spec only allows subfields with the group and nested fields, so any workaround based on the use of group-nested would not work at the moment.

So we need to align the behavior of Fleet, package-spec, and the current assumptions in existing integrations. Probably doing the following:

• Assume that nested and group-nested are synonyms, even if they were not originally, we are in a situation now where nested is used as group-nested, and group-nested is not used.
• Replace implementation of nested with the implementation of group-nested in Fleet, so they effectively behave the same.
• Allow in package spec the use of group-nested in all cases where nested is allowed, this will allow to fix issues in packages for current versions of packages.
  • Allow to use group-nested wherever nested can be used #789
• There are two additional parameters for nested objects called include_in_parent and include_in_root, we have only found uses of them in tests, and not on any actual package. We will need to confirm its expected behaviour and decide if we need to keep them.
  • Decided to keep them if possible, just in case they are needed, but no integration is using them at the moment.
• Consider replacing nested with group-nested in affected packages.

Update: After further investigation the assumptions on group-nested were not correct. group-nested is only used internally in Kibana, and it is fine if it continues this way. So we won't change this. Plan changes to:

• Fix generation of template for subfields of type: nested in Fleet.
  • [Fleet] Expand subfields of nested objects when generating template kibana#191730
• For packages supporting older versions of Fleet, they can use the syntax with one field per definition, like here.

Thanks @mrodm for finding that some mappings were not being applied!