[Exchange Online Message Trace] Add Additional Look-back Time & Fix Cursor Value #8717

BenB196 · 2023-12-13T01:53:48Z

Enhancement & Bug

Proposed commit message

This PR makes two changes to the Exchange Online Message Trace integration:

1. Performs a refactor of httpjson.yml.hbs file to move the request.* & response.* parts together, or inversely moving all the fields/tags stuff towards the end of the file. (making it easier to follow from my perspective)

Fixes an issue with how the value for last_execution_datetime cursor was being set.
- Previously, the cursor value was using now, the issue with this is that the cursor sets it value once "all events ... are published". This meant that if a cycle took five minutes to complete, there could be a five minute gap in time/data between the last iteration's EndDate and the next iteration's StartDate. This appears to have always been an "issue" but became more noticeable with [integrations][Exchange Online Message Trace] - Refactored the integration and reworked the documentation #8550 where it changed the default interval from 1m to 1h, making it take far longer to process an iteration and making the gap in data more noticable.
- ~~To "fix" this issue, rather than using now, we take the .last_event.EndDate as the value. This should ensure that the next iteration always starts where the last one ended.~~
- Side Note: This also fixes another issue, for if Microsoft ever has an issue publishing Message Traces. Before this change, the integration would continue on as if there was no problem and leave a potentially large gap in data. Now the integration will hold at the last published event and pick up once Microsoft fixes the event publishing.
Adds the ability to configure an additional look-back time to ensure data is properly collected.
- See: [Exchange Online Message Trace] Add Additional Look-back Time & Fix Cursor Value #8717 (comment)

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

I ran elastic-package.exe test and the tests passed, however, I'm honestly not too familiar with the httpjson input. Would someone be able to take a look and confirm that this does indeed look correct, or if there is any additional testing that can be done? Looking at how some of the other cursors do it; https://github.com/search?q=repo%3Aelastic%2Fintegrations+%22cursor%3A%22&type=code this does appear to be "correct".
There is a potential "bug" here in that if last_execution_datetime ever gets too far out of date (older than Microsoft's Message Trace retention duration - 90 days), then the integration would be "stuck" in this state until recreated. This seems like it would be a similar issue with a number of other integrations, so I opted not to try and solve it here as I'm not sure how "real" of a "bug" this actually is.

How to test this PR locally

I ran elastic-package.exe test and everything passed.
I validated that the integration yaml updated correctly in Kibana/Fleet.

Related issues

Closes [Exchange Online Message Trace] Split Polling Interval into two settings, Polling Interval & Lookback Interval #8674
- Found the cursor issue while investigating this issue.
Relates [integrations][Exchange Online Message Trace] - Refactored the integration and reworked the documentation #8550

Screenshots

(Interval for the screenshots set to 5m for easier visibility into the change)

Current/main/1.17.x:

This PR:

elasticmachine · 2023-12-13T01:57:34Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-12-20T02:47:03.949+0000
Duration: 17 min 51 sec

Test stats 🧪

Test	Results
Failed	0
Passed	8
Skipped	0
Total	8

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

packages/microsoft_exchange_online_message_trace/data_stream/log/agent/stream/httpjson.yml.hbs

elasticmachine · 2023-12-13T09:15:33Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

This reverts commit f804e26.

BenB196 · 2023-12-17T23:30:36Z

I've updated the PR to include the ability to configure an additional look-back time. I chose 1h as the default value here, somewhat arbitrarily, but during testing I found that this was generally "good enough" without hitting the negative side-effects of going with some value like 24h.

I've updated the docs to include a mention this new field and the potential for missing events.

Unfortunately, Microsoft doesn't seem to offer a way to sort by a similar field like event.ingested, so need to leverage a raw look-back here and then dedupe events. (Deduping is already a part of this integration).

I've updated the main PR description with screenshots showing the difference between current and this PR.

I took the "wording" for this new field mainly from Kibana's Security Rules "Additional Look-back time" field.

ShourieG · 2023-12-18T04:24:40Z

/test

packages/microsoft_exchange_online_message_trace/docs/README.md

packages/microsoft_exchange_online_message_trace/_dev/build/docs/README.md

elasticmachine · 2023-12-18T04:37:05Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine · 2023-12-18T04:39:58Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`1/1`)	💚
Classes	100.0% (`1/1`)	💚
Methods	100.0% (`16/16`)	💚 75.0
Lines	98.12% (`261/266`)	👎 -1.88
Conditionals	100.0% (`0/0`)	💚

ShourieG · 2023-12-18T04:42:04Z

@BenB196 The change looks pretty good so far barring a few documentation issues, it would be awesome if you could add some system tests to simulate this updated behaviour of the integration.

Co-authored-by: ShourieG <[email protected]>

BenB196 · 2023-12-18T11:35:15Z

@ShourieG, docs changes applied, regarding the system tests. I'm a bit stuck here. I'd like to have a system test which queries the test endpoint, and asserts 2 (as it today). Then the test endpoint adds a 3rd entry and the agent queries it again and should assert 3. The issue I'm having is I can't seem to figure out how to have the system test-httpjson-config.yml run two queries and assert two different values for each run.

Would you happen to know of a similar system test example or point to the docs which highlight how to implement a multi-run system test?

efd6 · 2023-12-18T21:31:53Z

@BenB196 You can have 2 system test configs, one for the 2 doc case and one for the 3 doc case.

BenB196 · 2023-12-18T22:04:03Z

@BenB196 You can have 2 system test configs, one for the 2 doc case and one for the 3 doc case.

Hi @efd6 thanks for the reply. Question, will both tests then use the same Elastic Agent state? The reason I ask is I'd like the system test if possible to follow a flow like:

Spin up system test
Publish 2 events
Elastic Agent Asserts 2
Publish 3rd event between the initial 2 events
Elastic Re-reads event with existing agent state, and asserts 3 events.

This test would allow for covering both the cursor and look-back changes.

efd6 · 2023-12-18T23:16:44Z

No, they would be separate tests. You would need to engineer the 2 and then 1 behaviour in the 3 event test. The 2 event test case would test the initial 2 events in the 3 event case.

BenB196

(This comment is partially outdated, see: #8717 (comment))

I've expanded on the system tests here a bit.

There is still an issue with this though that I'll need to take a look at tomorrow, but pushing in the event someone else wants to take a look.

The current issue is that the cursor

cursor:
  last_execution_datetime:
    value: '[[ toJSON .last_event.message "EndDate" ]]'

Can become null/empty, in the event that the .last_response is empty, more specifically, you get value: [] (which it can be), and defaults back to now. I'm not sure the best way to handle this issue currently.

A few options I see:

Default to now.
- This is the easiest, but has the disadvantage of bringing back some of the issues this PR initially tried to solve.
Find a way to parse the $filter param to try and get the EndDate value from it.
- This would provide the current value, but I don't see a way to easily parse this param to get the value within the httpjson templating.
Open to ideas/suggestions that people might have.

...soft_exchange_online_message_trace/data_stream/log/_dev/test/system/test-httpjson-config.yml

packages/microsoft_exchange_online_message_trace/data_stream/log/agent/stream/httpjson.yml.hbs

...soft_exchange_online_message_trace/data_stream/log/elasticsearch/ingest_pipeline/default.yml

packages/microsoft_exchange_online_message_trace/_dev/deploy/docker/files/config.yml

efd6 · 2023-12-19T03:20:15Z

/test

...online_message_trace/data_stream/log/_dev/test/system/test-httpjson-null-trailing-config.yml

packages/microsoft_exchange_online_message_trace/data_stream/log/agent/stream/httpjson.yml.hbs

...soft_exchange_online_message_trace/data_stream/log/elasticsearch/ingest_pipeline/default.yml

efd6 · 2023-12-20T02:46:49Z

/test

Co-authored-by: Dan Kortschak <[email protected]>

packages/microsoft_exchange_online_message_trace/data_stream/log/agent/stream/httpjson.yml.hbs

…og/agent/stream/httpjson.yml.hbs Co-authored-by: Dan Kortschak <[email protected]>

efd6 · 2023-12-21T02:28:58Z

/test

efd6

LGTM, but waiting for @ShourieG.

ShourieG

LGTM

elasticmachine · 2023-12-21T03:02:03Z

Package microsoft_exchange_online_message_trace - 1.18.0 containing this change is available at https://epr.elastic.co/search?package=microsoft_exchange_online_message_trace

elasticmachine · 2023-12-21T03:02:04Z

Package microsoft_exchange_online_message_trace - 1.18.0 containing this change is available at https://epr.elastic.co/search?package=microsoft_exchange_online_message_trace

* upstream/main: (117 commits) [TI MISP] Add IOC expiration support (#8639) Add CSPM Rules 6.2, 6.3 and 6.4 (#8778) [Infoblox NIOS] Update timestamp parsing logic (#8767) [Rapid7 InsightVM] Split vulnerability categories into array (#8768) [Exchange Online Message Trace] Add Additional Look-back Time & Fix Cursor Value (#8717) [Buildkite] Update bucket settings (#8765) Remove Jenkins .ci folder (#8766) First part of removal of Jenkins jobs (#8763) misp: parse URIs for URI type threats (#8760) [amazon_security_lake] Added support for all the OCSF Classes (#8579) [Buildkite] Update settings for integrations pipeline (#8758) [TI ThreatQ] Add IOC expiration support (#8691) [ti_opencti] Support OpenCTI 5.12 by removing filters parameter (#8744) [Cribl] Updating setup guidance for Cribl field (#8746) crowdstrike: add userinfo enrichment support and map fields to ECS (#8742) [etcd] Enable TSDB for metrics datastream (#8649) Bump golang.org/x/crypto from 0.16.0 to 0.17.0 (#8749) auditd: relax field_split pattern and handle AVC header (#8748) Update cloud packages codeowner (#8672) [O11Y] [AWS Billing] Convert "Total Estimated Charges" visualization to new metric (#8509) ...

BenB196 added 2 commits December 12, 2023 20:11

refactor order of httpjson file config

f804e26

fix cursor

d0fdb5f

BenB196 requested a review from a team as a code owner December 13, 2023 01:53

Update changelog.yml

f912540

efd6 reviewed Dec 13, 2023

View reviewed changes

packages/microsoft_exchange_online_message_trace/data_stream/log/agent/stream/httpjson.yml.hbs Show resolved Hide resolved

packages/microsoft_exchange_online_message_trace/data_stream/log/agent/stream/httpjson.yml.hbs Outdated Show resolved Hide resolved

narph added the Team:Security-External Integrations label Dec 13, 2023

Revert "refactor order of httpjson file config"

e5ee442

This reverts commit f804e26.

BenB196 changed the title ~~[Exchange Online Message Trace] Fix Cursor Value and Slight Refactor~~ [Exchange Online Message Trace] Fix Cursor Value Dec 13, 2023

add additional look-back time

aff473a

BenB196 changed the title ~~[Exchange Online Message Trace] Fix Cursor Value~~ [Exchange Online Message Trace] Add Additional Look-back Time & Fix Cursor Value Dec 17, 2023

BenB196 requested review from ShourieG and efd6 December 17, 2023 23:31

fix doc wording

817519c

ShourieG reviewed Dec 18, 2023

View reviewed changes

packages/microsoft_exchange_online_message_trace/docs/README.md Outdated Show resolved Hide resolved

ShourieG reviewed Dec 18, 2023

View reviewed changes

packages/microsoft_exchange_online_message_trace/_dev/build/docs/README.md Outdated Show resolved Hide resolved

ShourieG reviewed Dec 18, 2023

View reviewed changes

packages/microsoft_exchange_online_message_trace/_dev/build/docs/README.md Outdated Show resolved Hide resolved

BenB196 and others added 2 commits December 18, 2023 06:21

Apply suggestions from code review

bff9eea

Co-authored-by: ShourieG <[email protected]>

Run build to update docs

76fb689

make existing system test, test skiptoken

6f14743

BenB196 added 3 commits December 18, 2023 21:56

more system tests

efdd3ef

revert unneeded change

7e5b3dc

update cursor

0b5d0ad

BenB196 commented Dec 19, 2023

View reviewed changes

BenB196 added 2 commits December 19, 2023 18:48

improve drop processor and add pipeline test

6ff67b8

fix cursor and add tests

7da5896

BenB196 commented Dec 20, 2023

View reviewed changes

...online_message_trace/data_stream/log/_dev/test/system/test-httpjson-null-trailing-config.yml Show resolved Hide resolved

packages/microsoft_exchange_online_message_trace/data_stream/log/agent/stream/httpjson.yml.hbs Show resolved Hide resolved

BenB196 requested a review from ShourieG December 20, 2023 02:35

efd6 reviewed Dec 20, 2023

View reviewed changes

...soft_exchange_online_message_trace/data_stream/log/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

BenB196 and others added 2 commits December 20, 2023 06:18

further refine drop processor condition

b7b7501

Co-authored-by: Dan Kortschak <[email protected]>

add comments about $filter param modification

e6cd256

BenB196 requested a review from efd6 December 20, 2023 23:54

efd6 reviewed Dec 21, 2023

View reviewed changes

packages/microsoft_exchange_online_message_trace/data_stream/log/agent/stream/httpjson.yml.hbs Outdated Show resolved Hide resolved

Update packages/microsoft_exchange_online_message_trace/data_stream/l…

86e6475

…og/agent/stream/httpjson.yml.hbs Co-authored-by: Dan Kortschak <[email protected]>

efd6 approved these changes Dec 21, 2023

View reviewed changes

ShourieG approved these changes Dec 21, 2023

View reviewed changes

ShourieG merged commit 7a12532 into elastic:main Dec 21, 2023

BenB196 deleted the microsoft-exchange-fixes branch December 21, 2023 19:34

andrewkroh added the Integration:microsoft_exchange_online_message_trac Microsoft Exchange Online Message Trace label Jul 22, 2024

chrisberkhout mentioned this pull request Jan 23, 2025

[microsoft_exchange_online_message_trace] Remove event.start/end #12446

Merged

5 tasks

[Exchange Online Message Trace] Add Additional Look-back Time & Fix Cursor Value #8717

[Exchange Online Message Trace] Add Additional Look-back Time & Fix Cursor Value #8717

Uh oh!

Conversation

BenB196 commented Dec 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Dec 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Dec 13, 2023

Uh oh!

BenB196 commented Dec 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ShourieG commented Dec 18, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Dec 18, 2023

🚀 Benchmarks report

Uh oh!

elasticmachine commented Dec 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

ShourieG commented Dec 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BenB196 commented Dec 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

efd6 commented Dec 18, 2023

Uh oh!

BenB196 commented Dec 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

efd6 commented Dec 18, 2023

Uh oh!

BenB196 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 commented Dec 19, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 commented Dec 20, 2023

Uh oh!

Uh oh!

efd6 commented Dec 21, 2023

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

ShourieG left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Dec 21, 2023

Uh oh!

elasticmachine commented Dec 21, 2023

Uh oh!

BenB196 commented Dec 13, 2023 •

edited

Loading

elasticmachine commented Dec 13, 2023 •

edited

Loading

BenB196 commented Dec 17, 2023 •

edited

Loading

elasticmachine commented Dec 18, 2023 •

edited

Loading

ShourieG commented Dec 18, 2023 •

edited

Loading

BenB196 commented Dec 18, 2023 •

edited

Loading

BenB196 commented Dec 18, 2023 •

edited

Loading

BenB196 left a comment •

edited

Loading