Add GCS input to Akamai

packages/akamai/_dev/build/docs/README.md

@@ -12,6 +12,17 @@ Use this API to get security event data generated on the ​Akamai​ platform a
 
 See [Akamai API get started](https://techdocs.akamai.com/siem-integration/reference/api-get-started) to set up your Akamai account and get your credentials.
 
+### To collect data from GCS Bucket [Beta], follow the below steps:
+- Configure the [Data Forwarder](https://techdocs.akamai.com/datastream2/docs/stream-google-cloud/) to ingest data into a GCS bucket.
+- Configure the GCS bucket names and credentials along with the required configs under the "Collect Akamai SIEM logs via Google Cloud Storage" section. 
+- Make sure the service account and authentication being used, has proper levels of access to the GCS bucket [Manage Service Account Keys](https://cloud.google.com/iam/docs/creating-managing-service-account-keys/)
+
+**Note**:
+- The GCS input currently does not support fetching of buckets using bucket prefixes, so the bucket names have to be configured manually for each data stream.
+- The GCS input currently only accepts a service account JSON key or a service account JSON file for authentication.
+- The GCS input currently only supports JSON data.
+- This input is still in beta.
+
 {{fields "siem"}}
 
 {{event "siem"}}

packages/akamai/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "2.3.0"
+  changes:
+    - description: Added support for GCS input.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/4728
 - version: "2.2.0"
   changes:
     - description: Update package to ECS 8.5.0.

packages/akamai/data_stream/siem/agent/stream/gcs.yml.hbs

@@ -0,0 +1,38 @@
+{{#if project_id}}
+project_id: {{project_id}}
+{{/if}}
+{{#if service_account_key}}
+auth.credentials_json.account_key: {{service_account_key}}
+{{/if}}
+{{#if service_account_file}}
+auth.credentials_file.path: {{service_account_file}}
+{{/if}}
+{{#if number_of_workers}}
+max_workers: {{number_of_workers}}
+{{/if}}
+{{#if polling}}
+poll: {{polling}}
+{{/if}}
+{{#if poll_interval}}
+poll_interval: {{poll_interval}}
+{{/if}}
+{{#if bucket_timeout}}
+bucket_timeout: {{bucket_timeout}}
+{{/if}}
+{{#if buckets}}
+buckets:
+{{buckets}}
+{{/if}}
+{{#if tags}}
+tags:
+{{#each tags as |tag|}}
+  - {{tag}}
+{{/each}}
+{{/if}}
+{{#contains "forwarded" tags}}
+publisher_pipeline.disable_host: true
+{{/contains}}
+{{#if processors}}
+processors:
+{{processors}}
+{{/if}}

packages/akamai/data_stream/siem/manifest.yml

@@ -3,6 +3,9 @@ title: Akamai SIEM Logs
 release: experimental
 streams:
   - input: httpjson
+    template_path: httpjson.yml.hbs
+    title: Akamai SIEM logs
+    description: Collect Akamai logs via the SIEM API
     vars:
       - name: api_host
         type: text
@@ -100,6 +103,100 @@ streams:
         required: false
         show_user: false
         description: "Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. \nThis executes in the agent before the logs are parsed. \nSee [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details.\n"
-    template_path: httpjson.yml.hbs
-    title: Akamai SIEM logs
-    description: Collect Akamai logs via the SIEM API
+  - input: gcs
+    title: Collect Akamai SIEM logs via Google Cloud Storage [Beta]
+    description: Collecting SIEM logs from Akamai via Google Cloud Storage.
+    template_path: gcs.yml.hbs
+    vars:
+      - name: project_id
+        type: text
+        title: "Project Id"
+        description: It is a required parameter to collect logs via GCS.
+        multi: false
+        required: true
+        show_user: true
+        default: my-project-id
+      - name: service_account_key
+        type: password
+        title: "Credentials json key"
+        description: It is an optional parameter for authentication.
+        multi: false
+        required: false
+        show_user: true
+      - name: service_account_file
+        type: text
+        title: "Credentials file path"
+        description: It is an optional parameter for authentication.
+        multi: false
+        required: false
+        show_user: false
+      - name: max_workers
+        type: integer
+        title: 'Maximum number of workers'
+        multi: false
+        required: false
+        show_user: true
+        default: 3
+        description: Determines how many workers are spawned per bucket.
+      - name: poll
+        type: bool
+        title: 'Polling'
+        multi: false
+        required: false
+        show_user: true
+        default: true
+        description: Determines if the bucket will be continuously polled for new documents.
+      - name: poll_interval
+        type: text
+        title: 'Polling interval'
+        multi: false
+        required: false
+        show_user: true
+        default: 15s
+        description: Determines the time interval between polling operations.
+      - name: bucket_timeout
+        type: text
+        title: 'Bucket Timeout'
+        multi: false
+        required: false
+        show_user: true
+        default: 120s
+        description: Defines the maximum time that the sdk will wait for a bucket api response before timing out.
+      - name: buckets
+        type: yaml
+        title: Buckets
+        description: "This attribute contains the details about a specific bucket like, name, max_workers, poll, 
+        poll_interval and bucket_timeout. The attribute 'name' is specific to a bucket as it describes the bucket name, 
+        while the fields max_workers, poll, poll_interval and bucket_timeout can exist both at the bucket level and at the global level. 
+        If you have already defined the attributes globally, then you can only specify the name in this yaml config. If you want to override any specific 
+        attribute for a specific bucket, then, you can define it here. Any attribute defined in the yaml will override the global definitions.
+        Please see the relevant [Documentation](https://www.elastic.co/guide/en/beats/filebeat/8.5/filebeat-input-gcs.html#attrib-buckets) for further information.\n"
+        required: true
+        show_user: true
+        default: |
+          # You can define as many buckets as you want here.
+          - name: siem_gcs_bucket_1
+          - name: siem_gcs_bucket_2
+          # The config below is an example of how to override the global config.
+          #- name: siem_gcs_bucket_3
+          #  max_workers: 3
+          #  poll: true
+          #  poll_interval: 10s
+          #  bucket_timeout: 30s
+      - name: processors
+        type: yaml
+        title: Processors
+        multi: false
+        required: false
+        show_user: false
+        description: |
+          Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details.
+      - name: tags
+        type: text
+        title: Tags
+        multi: true
+        required: true
+        show_user: false
+        default:
+          - forwarded
+          - akamai-siem  

packages/akamai/docs/README.md

@@ -12,6 +12,17 @@ Use this API to get security event data generated on the ​Akamai​ platform a
 
 See [Akamai API get started](https://techdocs.akamai.com/siem-integration/reference/api-get-started) to set up your Akamai account and get your credentials.
 
+### To collect data from GCS Bucket [Beta], follow the below steps:
+- Configure the [Data Forwarder](https://techdocs.akamai.com/datastream2/docs/stream-google-cloud/) to ingest data into a GCS bucket.
+- Configure the GCS bucket names and credentials along with the required configs under the "Collect Akamai SIEM logs via Google Cloud Storage" section. 
+- Make sure the service account and authentication being used, has proper levels of access to the GCS bucket [Manage Service Account Keys](https://cloud.google.com/iam/docs/creating-managing-service-account-keys/)
+
+**Note**:
+- The GCS input currently does not support fetching of buckets using bucket prefixes, so the bucket names have to be configured manually for each data stream.
+- The GCS input currently only accepts a service account JSON key or a service account JSON file for authentication.
+- The GCS input currently only supports JSON data.
+- This input is still in beta.
+
 **Exported fields**
 
 | Field | Description | Type |

packages/akamai/manifest.yml

@@ -1,6 +1,6 @@
 name: akamai
 title: Akamai
-version: "2.2.0"
+version: "2.3.0"
 release: ga
 description: Collect logs from Akamai with Elastic Agent.
 type: integration
@@ -22,5 +22,8 @@ policy_templates:
       - type: httpjson
         title: "Collect Akamai SIEM logs via API"
         description: "Collecting SIEM logs from Akamai via API"
+      - type: gcs
+        title: "Collect Akamai SIEM logs via Google Cloud Storage"
+        description: "Collecting SIEM logs from Akamai via Google Cloud Storage"
 owner:
   github: elastic/security-external-integrations