[Akamai] Use 'offset' value as cursor #5827
Conversation
The Akamai SIEM API returns an 'offset' value that encodes the storage time of the last returned event. Use offset as the cursor value when making API requests. This assumes that the API response always includes an offset. And that the response has total == 0 when there is no more data available. The version is marked as beta because it has not been tested against the real Akamai service. Relates: #5826
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
🤖 GitHub commentsExpand to view the GitHub comments
To re-run your PR in the CI, just comment with:
|
|
Pinging @elastic/security-external-integrations (Team:Security-External Integrations) |
🌐 Coverage report
|
| categories: [security, cdn_security] | ||
| conditions: | ||
| kibana.version: "^8.5.0" | ||
| kibana.version: "^8.4.0" |
There was a problem hiding this comment.
I dropped this back down to allow users on 8.4.0 to test this. The http request tracer feature requires 8.5.0, so it won't take effect if you enable it under 8.4.0 (the config option is ignored by the older input).
| value: "[[.cursor.last_execution_datetime]]" | ||
| default: '[[ (now (parseDuration "-{{initial_interval}}")).Unix ]]' | ||
| value: >- | ||
| [[ if not (index .cursor "last_offset") ]][[ (now (parseDuration "-24h")).Unix ]][[ end ]] |
There was a problem hiding this comment.
I see we are removing default: '[[ (now (parseDuration "-{{initial_interval}}")).Unix ]]', which makes this field in the manifest remain unused. Would we want to provide a condition such that if {{initial_interval}} is not specified then use [[ if not (index .cursor "last_offset") ]][[ (now (parseDuration "-24h")).Unix ]][[ end ]] otherwise use [[ (now (parseDuration "-{{initial_interval}}")).Unix ]] ?
There was a problem hiding this comment.
I meant to keep the {{initial_interval}}. I will put that back.
I was developing in Filebeat so I needed to get rid of the variable.
There was a problem hiding this comment.
Please take another look.
|
@andrewkroh Only one query here otherwise looks LGTM, similar to what I was thinking . |
|
Package akamai - 2.6.1-beta containing this change is available at https://epr.elastic.co/search?package=akamai |
|
My customer did not have the ability to flip a toggle for beta integrations (version 8.4.3). I was able to install it via running this in Kibana > Dev tools, just passing along (not sure if force is needed) |
What does this PR do?
The Akamai SIEM API returns an 'offset' value that encodes the storage time of the last returned event. Use offset as the cursor value when making API requests.
This assumes that the API response always includes an offset. And that the response has total == 0 when there is no more data available.
The version is marked as beta because it has not been tested against the real Akamai service.
Relates: #5826
Checklist
changelog.ymlfile.Related issues
Screenshots
This is a beta release. So users will need to select the version explicitly to test it.
Logs / Testing
These are the raw request / responses from testing with the mock API.
This was the contents of the Filebeat registry at the end of the test. You can see that
offset2is persisted.