m365_defender: Improve ECS Mappings for Detection Rules

[kcreddy]

Proposed commit message

A followup of PR https://github.com/elastic/integrations/pull/9860 
to add/enhance ECS mappings for detection rules.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

elastic-package stack down && elastic-package build && elastic-package stack up -d -v && eval "$(elastic-package stack shellinit)" && elastic-package test pipeline --generate -v

--- Test results for package: m365_defender - START ---
╭───────────────┬─────────────┬───────────┬─────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE       │ DATA STREAM │ TEST TYPE │ TEST NAME                           │ RESULT │ TIME ELAPSED │
├───────────────┼─────────────┼───────────┼─────────────────────────────────────┼────────┼──────────────┤
│ m365_defender │ alert       │ pipeline  │ test-alert.log                      │ PASS   │   3.322375ms │
│ m365_defender │ event       │ pipeline  │ test-alert.log                      │ PASS   │   2.456875ms │
│ m365_defender │ event       │ pipeline  │ test-app-and-identity.log           │ PASS   │   3.136125ms │
│ m365_defender │ event       │ pipeline  │ test-device.log                     │ PASS   │  51.277041ms │
│ m365_defender │ event       │ pipeline  │ test-email.log                      │ PASS   │   3.302375ms │
│ m365_defender │ incident    │ pipeline  │ test-incident.log                   │ PASS   │  10.775042ms │
│ m365_defender │ log         │ pipeline  │ test-m365-defender-empty-ndjson.log │ PASS   │   1.258417ms │
│ m365_defender │ log         │ pipeline  │ test-m365-defender-ndjson.log       │ PASS   │   4.567333ms │
╰───────────────┴─────────────┴───────────┴─────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: m365_defender - END   ---
Done

            "registry": {
                "data": {
                    "type": "Binary"
                },
+                "hive": "HKLM",
                "key": "SYSTEM\\ControlSet001\\Control\\WMI\\Security",
-                "path": "SYSTEM\\ControlSet001\\Control\\WMI\\Security\\782655b2-0575-4aa2-82b8-7fd560afeff6",
+                "path": "HKLM\\SYSTEM\\ControlSet001\\Control\\WMI\\Security\\782655b2-0575-4aa2-82b8-7fd560afeff6",
                "value": "782655b2-0575-4aa2-82b8-7fd560afeff6"
            },

Related issues

• Closes M365 Defender Integration: Rather than removing HKEY_CURRENT_USER and and HKEY_LOCAL_MACHINE, replace them with HKCU and HKLM to avoiding breaking detection rules #10143

Screenshots

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[chrisberkhout]

The change looks fine.

It seems there are many non-deprecated rules that use hive abbreviations like HKLM:
repo:elastic/detection-rules language:TOML NOT path:/^rules\/_deprecated\// HKLM.

Should those all change?

cc @bczifra, @w0rk3r

[kcreddy]

It seems there are many non-deprecated rules that use hive abbreviations like HKLM:

@chrisberkhout, thanks for noticing these rules. Thats quite a few rules containing HKLM. While a similar query for HKEY_LOCAL_MACHINE containing rules: repo:elastic/detection-rules HKEY_LOCAL_MACHINE language:TOML NOT path:/^rules/_deprecated// did not yield many results and nothing on registry.path.

I am now inclined to make the registry.path use the abbreviated HKLM instead of HKEY_LOCAL_MACHINE.

[kcreddy]

@chrisberkhout @w0rk3r I updated registry.path with abbreviated registry.hive and also derived host.os.type based on event.category. Please feel free to review.

[jamiehynds]

@kcreddy @w0rk3r Microsoft have recently rebranded M365 Defender to Microsoft XDR. We'll likely deprecate the current integration and move towards a new Microsoft XDR integration, where the custom fields will use microsoft_xdr. We'll reuse the existing M365D pipelines to preserve existing ECS mappings, but adjust custom field names, datastreams, etc.

We haven't started down that path yet, but want to get it on your radar to ensure detection rules will work with the current M365 Defender integration and future XDR integration.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: bb181ee

History

• 💔 Build #12766 failed 88b8a04
• 💚 Build #12707 succeeded 45eb90c
• 💚 Build #12704 succeeded 606eea5
• 💚 Build #12658 succeeded b291386
• 💔 Build #12631 failed eca0668

cc @kcreddy

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
94.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[w0rk3r]

@jamiehynds so the data continue to be the same in the new integration, right? Do we have an estimate on when we are going to work on this?

[w0rk3r]

This will only take effect if the above one is null, right?

[kcreddy]

yes, so this processor only executes when if: ctx.registry?.value == null, i.e., when registry.value is not set from above.

[jamiehynds]

@jamiehynds so the data continue to be the same in the new integration, right? Do we have an estimate on when we are going to work on this?

We'll still collect the same data, but the custom fields would be different - using microsoft_xdr.* instead of the current m365_defender

No time frame yet as work has yet to start, but @cpascale43 will be working on an issue and identify additional data we can pull from Microsoft XDR on top of what we already collect.

[elasticmachine]

Package m365_defender - 2.14.0 containing this change is available at https://epr.elastic.co/search?package=m365_defender