[aws] Add S3 option to s3access logs

[kaiyan-sheng]

Proposed commit message

This PR is to add reading logs directly from S3 as an option for s3access data stream. There are more data streams need this change, will add it later in the same PR.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Screenshots

I tested it with a sample log file just with content test1. It's not an s3acecss log so it shows it cant be parsed. But the message got ingested from s3 to elastic:

The agent policy looks like this:

inputs:
  - id: aws-s3-s3-e01b9c5e-c872-485d-81f3-d75a6ef534c9
    name: aws-1
    revision: 2
    type: aws-s3
    use_output: default
    meta:
      package:
        name: aws
        version: 3.0.0
    data_stream:
      namespace: default
    package_policy_id: e01b9c5e-c872-485d-81f3-d75a6ef534c9
    streams:
      - id: aws-s3-aws.s3access-e01b9c5e-c872-485d-81f3-d75a6ef534c9
        data_stream:
          dataset: aws.s3access
          type: logs
        bucket_arn: 'arn:aws:s3:::test-aws-s3-input-ks'
        number_of_workers: 5
        bucket_list_interval: 1m
        access_key_id: '${SECRET_0}'
        secret_access_key: '${SECRET_1}'
        session_token: '${SECRET_2}'
        tags:
          - forwarded
          - aws-s3access
        publisher_pipeline.disable_host: true

[elasticmachine]

🚀 Benchmarks report

Package aws 👍(8) 💚(4) 💔(5)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
inspector	1805.05	1438.85	-366.2 (-20.29%)	💔
securityhub_findings	1102.54	900.09	-202.45 (-18.36%)	💔
waf	6849.32	5263.16	-1586.16 (-23.16%)	💔
apigateway_logs	11363.64	4065.04	-7298.6 (-64.23%)	💔
cloudwatch_logs	500000	250000	-250000 (-50%)	💔

To see the full report comment with /test benchmark fullreport

[zmoog]

LGTM

[MichaelKatsoulis]

LGTM

[agithomas]

LGTM!

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: f3591d4

History

• 💚 Build #13746 succeeded 378b40e
• 💔 Build #13741 failed 16b9d59
• 💚 Build #13677 succeeded 4758cdb
• 💔 Build #13634 failed 6b66136
• 💚 Build #13576 succeeded c352776

cc @kaiyan-sheng

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
97.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

Package aws - 2.20.0 containing this change is available at https://epr.elastic.co/search?package=aws