[Akamai] Possible pagination issues #5826
Description
We are investigating possible issues with how pagination is implemented. This is based on my reading of the configuration and API docs only. This integration was developed by the community. Testing of the integration is based on API mocking from the original community contribution.
The httpjson configuration for the Akamai SIEM API uses time-based mode for the first request then switches to offset mode to read subsequent data until no more offset key is returned. Events are returned in the order of their storage time in the database so you should not use event timestamps in to/from query params.
At the end of an iteration the integration stores (now - 1m) as the cursor timestamp under the assumption that the data returned included everything that was stored in the DB up to now. To mitigate race conditions it uses the -1m look back window. Because that creates an overlap in time ranges it uses the HTTP request ID to prevent duplicate events from being indexed.
There are some assumptions about the API behavior that need to be checked. I have opened an enhancement (#5825) to enable raw request/response logging so we can observe the responses.
I have a few theories on potential issues.
Theory 1
The pagination loop never exits.
Questions
What stops pagination? Does the last response containing zero events not have an "offset" key in the ResponseContext? That looks like the only way that pagination would stop.
Theory 2
Data loss occurs when requests (including pagination) takes more than one minute.
The initial time based request includes a "to" param that is (now - 1m). Then it paginates over the response data until there is no more. (It is unknown if the pagination is up to the initially requested "to" point or if it returns data until there is no more). Under the assumption the API is returning data up to the initially requested "to" point, then having the httpjson input resume from (now - 1m) is wrong because the value of (now - 1m) at the time of the initial request is different than the value of (now - 1m) at the end of pagination. This is particularly true if the initial set of requests takes a long time period (like more than 1m).