Apache ingest pipeline doesn't tolerate existing "event.original" field

[jsvd]

From the apache ingest pipeline (link):

  - rename:
      field: message
      target_field: event.original

The rename processor documentation states:

If the field doesn’t exist or the new name is already used, an exception will be thrown.

This means that any document that already has an "event.original" field (with or without a "message") field will cause an ingestion error:

"message": "......",
"error": {
"message": "field [event.original] already exists"
},

A suggestion is to tolerate the presence of "event.original" and "message" fields by including an if condition in the rename processor.

[lalit-satapathy]

Hi @jsvd,

Is the issue only for "apache" integration. What happens for the other integrations which also renames "message" to "event.original"?

[ishleenk17]

@lalit-satapathy : This issue will be for integrations which are using rename but are not using either of the below:

1. ignore_failure: true
2. if ctx.event.original == null check

[ishleenk17]

LS will be resolving this issue with : https://github.com/logstash-plugins/logstash-input-elastic_agent/issues/3

[jsvd]

Following up here: Logstash will work on ensuring the Logstash's elastic_agent input doesn't modify data coming from Elastic Agent.
That said we'd still like to encourage ingest pipelines to be more tolerant to existing event.original fields and overall other rename operations. Any thoughts on proceeding with #3464?

[jsvd]

Some quick greps show that there are currently 213 pipelines with a rename processor that moves "message" to "event.original":

~/elastic/integrations
❯ grep -r -A 3 "rename:" . | grep -A 2 " field: message" | grep  "target_field: event.original" |  wc -l
     213

More than half of these skip the rename if the "message" field exists through the use of ignore_missing: true.

~/elastic/integrations main
❯ grep -r -A 3 "rename:" . | grep -A 2 " field: message" | grep -A 2 "target_field: event.original" | grep "ignore_missing: true" | wc -l
     132

Having only the ignore_missing enabled allows pipelines to receive data with just the "event.original", but will still crash with both fields exist.

A few already are protected against the existence of "event.original" through the use of a conditional checking if "event.original" isn't null:

❯ grep -r -A 5 "rename:" . | grep -A 3 " field: message" | grep -A 3 "target_field: event.original" | grep "if.*event.*original.*null"  | wc -l
       8

@ishleenk17 do you think it's worth broadening the scope of this issue or create a new one to standardize how integrations pipelines handle "message" and "event.original"?

[philippkahr]

We really need to find a way to have a standardised setting. We ran into this using Logstash, which adds the event.original field since 8. Thus making the Fortinet pipeline unusable as it does not allow failure for the set event.original processor.

Maybe we can find a way to leverage some form of tests that make sure that a skeleton of the integrations looks the same? This also goes a long way to the settings that every integration should offer, such as tags, processors...

[GalchukRuth]

Hi,
Struggling with the same problem in AWS module.
We see in logs field message and also event.original.
With the same error: "field [event.original] already exists"

Filebeat version 8.5

When will the problem be fixed?

[ishleenk17]

@jsvd can share the ETA for https://github.com/logstash-plugins/logstash-input-elastic_agent/issues/3. this looks to be important to avoid any further SDH's.

[jsvd]

hi folks, on our side we're close to merging logstash-plugins/logstash-input-beats#464 (comment) which will allow disabling several kinds of enrichments, including the event.original.

[jsvd]

Starting with Logstash 8.7.0, the Agent/Beats inputs now support an enrich => setting so that data taken in as is from Beats. More on this setting here: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html#plugins-inputs-beats-enrich. An example configuration, taken from the documentation:

input {
  beats {
    port => 5044
    enrich => none
  }
}

This closes the loop on the Logstash side, I believe there's still a goal to resolve the inconsistent message/event.original handling across integrations hinted at in #3451 (comment)