[ECS] [TSDB] Centralisation of Dimension Fields

[agithomas]

Scope

• Identification of common fields in package that can be moved to ecs as dimensions.
• Aggregation of fields that related to service integration and cloud native , if needed.
• Identification of ecs repo changes and ecs testing.
• Creation of PR

[agithomas]

From the context of packages having ownership of service integration team, there exist certain fields that are part of every package and they are potential candidates of becoming dimension fields.

• service.address
• host.ip or host.name (Expect duplicates if the host.ip is not public and is a part of a subnet)
• agent.id

[agithomas]

When metrics are collected from a resource running in cloud or in a container, below mentioned fields are potential candidates of becoming dimension fields

• cloud.instance.id or cloud.instance.name
• cloud.provider : This is considering multi-cloud / hybrid infra deployment
• cloud.project.id : One organisation can have multiple projects. This is to support multi-regional deployment
• container.id : container.name may not be apt here. There may exist pods having same container names running in same host.
• host.name: Expect duplicates if the host.ip is not public and is a part of a subnet
• agent.id

Should subnet / network name be include ? TBD

[ruflin]

• For apps running in k8s, we do wee something like k8s.cluster.name or similar?
• Lets assume we specify all the fields above in ECS as dimension fields. In the k8s case, the cloud provider fields would likely not be used. Do this also count then to the max 16 fields even if there are no values? And how does this look, if the fields are set in dynamic templates? (@P1llus )

[lalit-satapathy]

CC @tommyers-elastic @gizas @felixbarny for any suggestions/comments on the TSDB ECS dimension fields. Once we close on these ECS fields, we can raise a PR for the same.

[felixbarny]

@kruskall has tried to define the dimensions for APM (elastic/apm-server#9730) but quickly hit the dimension limit (elastic/elasticsearch#93564).

[ruflin]

I like the direction elastic/elasticsearch#93564 is taking. An initial shortcut might be to just increase the limit to 32 which could already help.

Looking at elastic/apm-server#9730, it seems there is some overlap with the dimensions proposed here but there are also quite a few dimensions which I would argue are potentially unique to apm data. It would be nice if all the default dimensions can be set in ECS (or a common base) but are only used / applied when there is actual data. This goes back to my question: Is the limit reached when a field for the dimension is there or if it is in the mapping itself. If mapping is already enough, will it help to have it in the dynamic template?

For each default dimension define, I would like to see us have a note on why it is a dimension. There went a lot of thought into which dimensions to pick and it should be persisted and shared.

[agithomas]

Can we consider the above mentioned list for Service Integration and for Cloud (#5193 (comment)). We may have to expect new common fields added to the above list as we test new scenario.

We know, in such cases, explicit dimension field mapping can be done to avoid being blocked.

Is the above list good enough to work towards preparing RFC in ECS ?

[agithomas]

• For apps running in k8s, we do wee something like k8s.cluster.name or similar?

I think, having host.ip and container.id will cover the criteria of unique identification of document. Don't you think so?

[ruflin]

Is the above list good enough to work towards preparing RFC in ECS ?

I assume so, you know best :-)

I think, having host.ip and container.id will cover the criteria of unique identification of document. Don't you think so?

Is a container.id globally unique? I assume the chances are low enough for conflicts to have this work.

[agithomas]

Is a container.id globally unique? I assume the chances are low enough for conflicts to have this work.

There are no references i could find that says a container.id repeats in a cluster.

I understand the scenario you are referring to - the scenario where multiple k8s cluster are provisioned. Cluster name is a good logical segregation in such cases.

Reference : https://cloud.google.com/stackdriver/docs/solutions/gke/observing

It may then be checked

• Are the ways to get the cluster name in GKE / EKS / Self managed installation different or same?
• Are these information currently captured ? If yes, which ecs field is currently used ?

These may be the questions that can be asked to the owners of GKE / EKS integration team.

[agithomas]

It may then be checked

• Are the ways to get the cluster name in GKE / EKS / Self managed installation different or same?
• Are these information currently captured ? If yes, which ecs field is currently used ?

These may be the questions that can be asked to the owners of GKE / EKS integration team.

My preference would be not to modify the ecs dimension fields based on an application or cluster technology / deployment architecture. Every technology such a aws.kinesis, gke , eks may have its own identifier a unique resource it is monitoring. In akw.kinesis it is aws.dimensions.streamName.

As part of integration enhacement to use TSDB, it is expected these unique fields that represent a resource is identified carefully as dimension field in the integration.

[agithomas]

data_stream.namespace is an important one too. Adding it to the confirmed list.