[multiple integrations] Lowercase host.name field #6057

LaZyDK · 2023-05-02T10:01:21Z

What does this PR do?

As defined in ECS 8.7, host.name is preferably lowercase.

It can contain what hostname returns on Unix systems, the fully qualified domain name (FQDN), or a name specified by the user. The recommended value is the lowercase FQDN of the host.

The below integrations have been edited to support this.

Cisco Secure Endpoint

Lowercase host.name

Cisco Umbrella

Lowercase host.name

PANW Cortex XDR

Lowercase host.name

PANW

Lowercase host.name

Fortinet Fortigate

Add event.type: denied when action is deny
Add host field
Lowercase host.name
Fix MAC address format
Add test

Microsoft DHCP

Lowercase host.name
remove dynamic_fields

Microsoft Defender Endpoint

Lowercase host.name
remove dynamic_fields

M365 Defender

Incident

Add tests
Add host fields
Copy deviceDnsName to host.name
Lowercase host.name

Alert:

Add host fields
Copy DeviceName to host.name in pipelines: alert, app and identity, device
Lowercase host.name

Log:

Add host fields
Copy deviceDnsName to host.name
Lowercase host.name

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

elasticmachine · 2023-05-02T10:04:52Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-05-24T08:11:27.659+0000
Duration: 18 min 51 sec

Test stats 🧪

Test	Results
Failed	0
Passed	129
Skipped	0
Total	129

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

LaZyDK · 2023-05-02T10:36:35Z

Ready for discussions and test

elasticmachine · 2023-05-03T11:32:53Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

kcreddy · 2023-05-05T08:09:34Z

/test

elasticmachine · 2023-05-05T08:28:10Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`10/10`)	💚
Files	100.0% (`33/33`)	💚
Classes	100.0% (`33/33`)	💚
Methods	95.139% (`274/288`)	👍 3.139
Lines	92.535% (`11453/12377`)	👎 -1.894
Conditionals	100.0% (`0/0`)	💚

packages/cisco_secure_endpoint/data_stream/event/elasticsearch/ingest_pipeline/default.yml

packages/fortinet_fortigate/data_stream/log/elasticsearch/ingest_pipeline/default.yml

...soft_defender_endpoint/data_stream/log/_dev/test/pipeline/test-defenderatp.log-expected.json

...point/kibana/dashboard/microsoft_defender_endpoint-65402c30-ca6a-11ea-9d4d-9737a63aaa55.json

packages/panw/data_stream/panos/elasticsearch/ingest_pipeline/globalprotect.yml

packages/panw_cortex_xdr/changelog.yml

Co-authored-by: Krishna Chaitanya Reddy Burri <[email protected]>

kcreddy · 2023-05-05T10:20:05Z

/test

LaZyDK · 2023-05-05T10:45:00Z

Well that's some awkward test results for panw.

I am getting this locally:

--- Test results for package: panw - START ---
╭─────────┬─────────────┬───────────┬──────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                    │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼──────────────────────────────────────────────┼────────┼──────────────┤
│ panw    │ panos       │ pipeline  │ test-panw-panos-authentication-sample.log    │ PASS   │   48.51725ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-config-sample.log            │ PASS   │    6.00225ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-correlated-events-sample.log │ PASS   │  15.830709ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-decryption-sample.log        │ PASS   │  18.005458ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-globalprotect-sample.log     │ PASS   │     34.052ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-gtp-sample.log               │ PASS   │   5.512625ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-hipmatch-sample.log          │ PASS   │   4.934625ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-inc-other-sample.log         │ PASS   │  17.633458ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-inc-threat-sample.log        │ PASS   │  85.674084ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-inc-traffic-sample.log       │ PASS   │  93.949667ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-inc-traffic.json             │ PASS   │  10.136167ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-ip-tag-sample.log            │ PASS   │   3.899834ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-sctp-sample.log              │ PASS   │   3.654917ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-system-sample.log            │ PASS   │   3.894459ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-threat-sample.log            │ PASS   │   103.6335ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-traffic-sample.log           │ PASS   │  91.255583ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-tunnel-inspection-sample.log │ PASS   │   5.151458ms │
│ panw    │ panos       │ pipeline  │ test-panw-panos-userid-sample.log            │ PASS   │  16.124375ms │
╰─────────┴─────────────┴───────────┴──────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: panw - END   ---
Done

--- Test results for package: panw - START ---
╭─────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ panw    │ panos       │ system    │ logfile   │ PASS   │ 38.838056667s │
│ panw    │ panos       │ system    │ tcp       │ PASS   │ 24.517048291s │
│ panw    │ panos       │ system    │ udp       │ PASS   │ 24.565975292s │
│ panw    │ panos       │ system    │ udp-tz    │ PASS   │ 24.312194666s │
│ panw    │ panos       │ system    │ tls       │ PASS   │ 27.659064584s │
╰─────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: panw - END   ---

LaZyDK · 2023-05-05T10:58:53Z

Found the error. Generating new test results and will commit when done.

LaZyDK · 2023-05-05T11:13:28Z

Ready for another test

efd6

Minor nits, then LGTM. Will also wait for approval from @jamiehynds.

packages/microsoft_defender_endpoint/data_stream/log/elasticsearch/ingest_pipeline/default.yml

…arch/ingest_pipeline/default.yml Co-authored-by: Dan Kortschak <[email protected]>

kcreddy · 2023-05-17T14:41:02Z

/test

efd6 · 2023-05-22T03:46:14Z

/test

jamiehynds · 2023-05-22T07:42:56Z

LGTM

kcreddy · 2023-05-23T09:14:38Z

/test

P1llus · 2023-05-24T07:34:26Z

Just as a note for the reviewers, this will temporarily start creating duplicate host entries, because hostnames that was not previously lowercase, will now become that, which will conflict with things like SIEM rules that aggregates on hostname etc, for a short time period. This is however a healthy change for the future of the users. Should get a thumbs up from @jamiehynds as well.

@P1llus when you say a short period of time, do we know how long any impacted hostnames (i.e. not currently lowercase) will remain an issue? It's short term pain for long term gain, but might be worth creating a brief article for Support, to get ahead of issues that arise from the change. FYI @jamesspi based on the SIEM impact.

Hard to say, depends on the usecase and how long its used:

Transforms without any maximum age.
ML trained models for entity analytics or anything host based.
Anything custom (rules, transforms etc).

I feel we should still merge, but its just good to know. The current status before we merge, is that it causes issues already due to the mix of lower/upper case, so its fixing much more.

P1llus · 2023-05-24T08:11:13Z

/test

kcreddy

LGTM

elasticmachine · 2023-05-24T09:15:01Z

Package cisco_secure_endpoint - 2.12.0 containing this change is available at https://epr.elastic.co/search?package=cisco_secure_endpoint

elasticmachine · 2023-05-24T09:20:11Z

Package cisco_umbrella - 1.12.0 containing this change is available at https://epr.elastic.co/search?package=cisco_umbrella

elasticmachine · 2023-05-24T09:24:53Z

Package fortinet_fortigate - 1.11.0 containing this change is available at https://epr.elastic.co/search?package=fortinet_fortigate

elasticmachine · 2023-05-24T09:29:30Z

Package m365_defender - 1.9.0 containing this change is available at https://epr.elastic.co/search?package=m365_defender

elasticmachine · 2023-05-24T09:34:30Z

Package microsoft_defender_endpoint - 2.11.0 containing this change is available at https://epr.elastic.co/search?package=microsoft_defender_endpoint

elasticmachine · 2023-05-24T09:39:46Z

Package microsoft_dhcp - 1.12.0 containing this change is available at https://epr.elastic.co/search?package=microsoft_dhcp

elasticmachine · 2023-05-24T09:45:30Z

Package panw - 3.8.0 containing this change is available at https://epr.elastic.co/search?package=panw

elasticmachine · 2023-05-24T09:50:42Z

Package panw_cortex_xdr - 1.10.0 containing this change is available at https://epr.elastic.co/search?package=panw_cortex_xdr

LaZyDK added 11 commits May 2, 2023 08:34

Remove empty events and anonymize test data

2e8a9b2

Merge branch 'elastic:main' into panw_xdr_empty_event

7464a1f

Fixup fields

da0d710

Bump version

4591fc7

Fix panw_cortex_xdr

03d68b8

Fix panw

abd197a

Fix cisco_secure_endpoint

39cf1fc

Fix cisco_umbrella

e854665

Fix fortinet_fortigate

c39f02b

Fix microsoft_dhcp

79d09a7

Fix microsoft_defender_endpoint

7123314

LaZyDK requested a review from a team as a code owner May 2, 2023 10:01

Bump versions

f295a73

Add system test for microsoft_dhcp

e1d0cc5

jamiehynds added the Team:Security-External Integrations label May 3, 2023

kcreddy reviewed May 5, 2023

View reviewed changes

LaZyDK and others added 3 commits May 5, 2023 12:00

Remove questionmark

f95698e

Co-authored-by: Krishna Chaitanya Reddy Burri <[email protected]>

Check for host.name

560d7d8

Invert host.name check

1fe4789

Commit panw tests

8073d97

Fixup panw and system tests

e1bb4da

efd6 reviewed May 16, 2023

View reviewed changes

packages/microsoft_defender_endpoint/data_stream/log/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

Update packages/microsoft_defender_endpoint/data_stream/log/elasticse…

2cf2e84

…arch/ingest_pipeline/default.yml Co-authored-by: Dan Kortschak <[email protected]>

Merge branch 'main' into ecs_host_name

179da36

Merge branch 'main' into ecs_host_name

ab95cf0

System test for correct elastic version

4bc53fb

kcreddy approved these changes May 24, 2023

View reviewed changes

kcreddy merged commit 8a0bb78 into elastic:main May 24, 2023

LaZyDK deleted the ecs_host_name branch May 24, 2023 09:05

[multiple integrations] Lowercase host.name field #6057

[multiple integrations] Lowercase host.name field #6057

Uh oh!

Conversation

LaZyDK commented May 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

Uh oh!

elasticmachine commented May 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

LaZyDK commented May 2, 2023

Uh oh!

elasticmachine commented May 3, 2023

Uh oh!

kcreddy commented May 5, 2023

Uh oh!

elasticmachine commented May 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kcreddy commented May 5, 2023

Uh oh!

LaZyDK commented May 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LaZyDK commented May 5, 2023

Uh oh!

LaZyDK commented May 5, 2023

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kcreddy commented May 17, 2023

Uh oh!

efd6 commented May 22, 2023

Uh oh!

jamiehynds commented May 22, 2023

Uh oh!

kcreddy commented May 23, 2023

Uh oh!

P1llus commented May 24, 2023

Uh oh!

P1llus commented May 24, 2023

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented May 24, 2023

Uh oh!

elasticmachine commented May 24, 2023

Uh oh!

elasticmachine commented May 24, 2023

Uh oh!

elasticmachine commented May 24, 2023

Uh oh!

elasticmachine commented May 24, 2023

Uh oh!

elasticmachine commented May 24, 2023

Uh oh!

elasticmachine commented May 24, 2023

Uh oh!

elasticmachine commented May 24, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

LaZyDK commented May 2, 2023 •

edited

Loading

elasticmachine commented May 2, 2023 •

edited

Loading

elasticmachine commented May 5, 2023 •

edited

Loading

LaZyDK commented May 5, 2023 •

edited

Loading