Restore and send gaps for Windows events #1747

duhminick · 2025-06-28T00:38:53Z

Description of the issue

Similar to #1721 but instead for Windows. With multi-threaded support, it is possible that certain events were not sent and the agent was shutdown: gaps in event delivery. We want to send those events on agent startup.

We also did not have any mocking for the sys calls to the Windows event API. This results in events being written to Windows directly since the actual sys calls were used for the tests.

Description of changes

Core Feature: Windows Event Log Gap Detection and Recovery

Enhanced wineventlog.go with gap processing capabilities
1. Added gapsToRead field to track missing event ranges during initialization and resubscription. gapsToRead is set by inverting the ranges from the restored state file.
2. Implemented gap prioritization logic in the main event loop - gaps are processed before new events
3. Added readGaps() method that opens separate query handles for each gap range and processes missing events
New range-based querying functionality
1. Added openAtRange() method to create targeted queries for specific event ID ranges
2. Implemented CreateRangeQuery() utility to build filter queries based on RecordID using the filter (EventRecordID > X and EventRecordID <= Y)
Architecture improvements for testability
1. Refactored Windows API calls into WindowsEventAPI interface to enable mocking
2. Consolidated system call functionality from sys_call.go into windows_event_api.go
3. Added SetEventAPI() method to inject mock implementations during testing
4. All Windows API calls now go through the interface (EvtSubscribe, EvtQuery, EvtNext, EvtClose, etc.)
5. Created mock_windows_event_api.go with full mock implementation of Windows Event Log APIs

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Unit tests
Tested locally

Manual Tests

No state

Given no state file and Windows event logs having records and the Windows event logs having records

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z (26 records in total, records 1-26).

Results in

The state file after update:

26
Windows-AlphabetLog_i-0be8d9fb5653d32a1_TwoAlphaLog
0-26

Gaps in range

Given the state file having the content

24
Windows-AlphabetLog_i-0be8d9fb5653d32a1_TwoAlphaLog
2-5,10-15,20-24

and the Windows event logs having records
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z AA AB AC (29 records in total).

Results in

Notice that the "already sent" events are not delivered again.

The state file after update:

29
Windows-AlphabetLog_i-0be8d9fb5653d32a1_TwoAlphaLog
0-29

Confirm tailing of event logs after agent stop

This is a continuation of the `Gaps in range` test. We want to verify that offset still works. So given the state of the `Gaps in range` test, we will stop the agent, add a new Windows event, then start the agent.

Results in

and the state file:

30
Windows-AlphabetLog_i-0be8d9fb5653d32a1_TwoAlphaLog
0-30

Confirm tailing of event logs while agent still running

Given the state file having the content

24
Windows-AlphabetLog_i-0be8d9fb5653d32a1_TwoAlphaLog
2-5,10-15,20-24

and the Windows event logs having records
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z (26 records in total).

results in

then once adding a Windows event using Write-EventLog -LogName $LogName -Source $Source -EventId 1027 -EntryType Information -Message "Alphabet Entry: Letter AA"

results in

with the state file having content:

27
Windows-AlphabetLog3_i-0be8d9fb5653d32a1_ThreeAlphaLog
0-27

Requirements

Before commiting your code, please do the following steps.

Run make fmt and make fmt-sh
Run make lint

Integration Tests

To run integration tests against this PR, add the ready for testing label.

duhminick · 2025-06-30T19:50:28Z

Seems like the Windows range-based offset is inclusive unlike the Linux one. Going to make an update for that.

Fix assertion in wineventlog test Fix mocks

plugins/inputs/windows_event_log/wineventlog/mock_windows_event_api.go

plugins/inputs/windows_event_log/wineventlog/wineventlog.go

plugins/inputs/windows_event_log/wineventlog/utils.go

jefchien · 2025-06-30T21:18:31Z

plugins/inputs/windows_event_log/wineventlog/utils.go

@@ -219,3 +238,17 @@ func insertPlaceholderValues(rawMessage string, evtDataValues []Datum) string {
 	}
 	return sb.String()
 }
+
+func utf16PtrToString(ptr *uint16) string {


nit: Seems like this is only used in the mock/tests. Consider moving it there.

I'm using it in two separate tests, utils_test.go and wineventlog_test.go - I figured it would be okay to have it here

If it's only used in tests, then IMO it should be in a test file.

jefchien · 2025-06-30T21:22:48Z

plugins/inputs/windows_event_log/wineventlog/utils.go

+	var levelsFilter string
+	formattedLevels := make([]string, len(levels))
+	for i, level := range levels {
+		formattedLevels[i] = fmt.Sprintf(eventLogLevelFilter, level)
+	}
+	if len(formattedLevels) > 0 {
+		levelsFilter = fmt.Sprintf("(%s)", strings.Join(formattedLevels, " or "))


nit: Could use a string.Builder instead of creating the slice and string. Not a big deal since this isn't called that frequently and there aren't that many levels/filters.

plugins/inputs/windows_event_log/wineventlog/utils.go

plugins/inputs/windows_event_log/wineventlog/wineventlog.go

jefchien · 2025-06-30T21:46:16Z

plugins/inputs/windows_event_log/wineventlog/utils.go

 	eventLogLevelFilter      = "Level='%s'"
 	eventIgnoreOldFilter     = "TimeCreated[timediff(@SystemTime) &lt;= %d]"
+	eventRangeFilter         = "EventRecordID &gt; %d and EventRecordID &lt; %d"


So this doesn't include the start or end record IDs?

Yeah, it won't include the start or end IDs. I used to have it include the start, but it would've led to one duplicate event

Should the upper bound be lte instead? From what I can tell, if the state file is

0-10,20-30

It means that 10 was sent, but 20 was not. So when we invert it to get the gaps, it would be (10, 20].

jefchien · 2025-06-30T22:06:47Z

plugins/inputs/windows_event_log/wineventlog/wineventlog.go

+				// Need to shift by 1 because the range logic assumes [start, end) whereas Windows events is [start, end]
+				if readingFromGap {
+					r.Shift(recordNumber + 1)
+				} else {
+					r.Shift(recordNumber)
+				}


Since this is just setting the number for the record that was just read, that means the end has to be inclusive. So each range that's sent is really just 1-wide and is actually just (start, end] and the equivalent of [end, end]. For example, if we just read ID 100, the range would look like (99, 100]. It represents knowing that 100 was processed, but not knowing it 99 was. This is fine when subscribing because of the EvtSubscribeStartAfterBookmark flag as mentioned above.

I'm not sure I understand why it won't work the same for the gaps. Isn't it going to try to restore from the state file that was generated during the subscribe? Is this just a matter of the RangeQuery not being upper bound inclusive?

jefchien · 2025-07-01T20:35:07Z

plugins/inputs/windows_event_log/wineventlog/wineventlog.go

+		handle, err := w.openAtRange(r)
+		defer func() {
+			winEventAPI.EvtClose(handle)
+		}()
+		if err != nil {
+			continue
+		}
+		readRecords := w.readFromHandle(handle)


nit: Right now, the defer call will happen at the end of readGaps. Consider moving this into a separate function so the close happens after each read.

func (w *windowsEventLog) readGap(r state.Range) []*windowsEventLogRecord

jefchien · 2025-07-01T20:41:40Z

plugins/inputs/windows_event_log/wineventlog/utils.go

@@ -219,3 +238,17 @@ func insertPlaceholderValues(rawMessage string, evtDataValues []Datum) string {
 	}
 	return sb.String()
 }
+
+func utf16PtrToString(ptr *uint16) string {


If it's only used in tests, then IMO it should be in a test file.

jefchien · 2025-07-01T20:49:31Z

plugins/inputs/windows_event_log/wineventlog/wineventlog_test.go

+func marshalRangeList(rl state.RangeList) string {
+	var marshalledRanges []string
+	for _, r := range rl {
+		text, _ := r.MarshalText()
+		marshalledRanges = append(marshalledRanges, string(text))
+	}
+	return strings.Join(marshalledRanges, ",")
+}
+
+func assertStateFileRange(t *testing.T, fileName string, rl state.RangeList) {
+	content, _ := os.ReadFile(fileName)
+	assert.Contains(t, string(content), marshalRangeList(rl))
+}


nit: Instead of marshaling, we could also just call the Restore function of the state manager.

jefchien · 2025-07-01T20:51:34Z

plugins/inputs/windows_event_log/wineventlog/wineventlog_test.go

+		// This is per EvtHandle hence the necessity to break up these calls
+		// 0, 1, 4 were "sent" previously (should be skipped)
+		mockAPI.AddMockEventsForQuery(createMockEventRecords(0, 2, 5))


nit: Think this is incorrect. Should be 1, 2, and 5 were previously sent and should be skipped.

jefchien · 2025-07-01T20:52:36Z

plugins/inputs/windows_event_log/wineventlog/wineventlog_test.go

+		elog.Init()
+
+		// Simulate new subscription events arriving
+		mockAPI.SimulateSubscriptionEvents(createMockEventRecords(5, 6, 7, 8))


nit: The subscription event should not have 5 in it if we follow our logic and 5 was already sent.

duhminick added 5 commits June 28, 2025 00:36

Restore and send gaps for Windows

2836691

Refactor sys_call to windows_event_api

61752ef

Fix test

e911076

Pull out duplicate function + fix assertion message

e974d86

Remove useless function

42ab528

duhminick changed the title ~~Dominic gap windows pr~~ Restore and send gaps for Windows events Jun 28, 2025

duhminick added the ready for testing Indicates this PR is ready for integration tests to run label Jun 30, 2025

duhminick marked this pull request as ready for review June 30, 2025 14:48

duhminick requested a review from a team as a code owner June 30, 2025 14:48

Changes to accomodate inclusve end offset

8bd9113

Fix assertion in wineventlog test Fix mocks

duhminick force-pushed the dominic-gap-windows-pr branch from 15061cd to 8bd9113 Compare June 30, 2025 21:32

jefchien reviewed Jun 30, 2025

View reviewed changes

duhminick added 2 commits June 30, 2025 23:08

Move mocked syscalls to test and updated mock usage

eb90d2f

Update query to use lte instead of lt

e4321ab

duhminick force-pushed the dominic-gap-windows-pr branch from a024adc to e4321ab Compare July 1, 2025 06:32

jefchien approved these changes Jul 1, 2025

View reviewed changes

Restore and send gaps for Windows events #1747

Are you sure you want to change the base?

Restore and send gaps for Windows events #1747

Uh oh!

Conversation

duhminick commented Jun 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of the issue

Description of changes

Core Feature: Windows Event Log Gap Detection and Recovery

License

Tests

Manual Tests

No state

Gaps in range

Confirm tailing of event logs after agent stop

Confirm tailing of event logs while agent still running

Requirements

Integration Tests

Uh oh!

duhminick commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jefchien Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

duhminick commented Jun 28, 2025 •

edited

Loading

duhminick commented Jun 30, 2025 •

edited

Loading

jefchien Jun 30, 2025 •

edited

Loading