Skip to content

[GCP] Add dataproc data stream #3789

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Sep 13, 2022
Merged

[GCP] Add dataproc data stream #3789

merged 7 commits into from
Sep 13, 2022

Conversation

gpop63
Copy link
Contributor

@gpop63 gpop63 commented Jul 21, 2022

What does this PR do?

Adds dataproc data stream.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@elasticmachine
Copy link

elasticmachine commented Jul 21, 2022

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-09-12T17:29:29.024+0000

  • Duration: 19 min 38 sec

Test stats 🧪

Test Results
Failed 0
Passed 80
Skipped 0
Total 80

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link

elasticmachine commented Jul 21, 2022

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (5/5) 💚
Files 100.0% (5/5) 💚 2.712
Classes 100.0% (5/5) 💚 2.712
Methods 91.579% (87/95) 👍 1.685
Lines 95.752% (1375/1436) 👍 4.698
Conditionals 100.0% (0/0) 💚

@gpop63 gpop63 marked this pull request as ready for review July 27, 2022 14:39
@gpop63 gpop63 requested review from a team as code owners July 27, 2022 14:39
@endorama endorama added Integration:gcp Google Cloud Platform Team:Cloud-Monitoring Label for the Cloud Monitoring team labels Aug 1, 2022
@gpop63
Copy link
Contributor Author

gpop63 commented Aug 8, 2022

/test


The `dataproc` dataset fetches metrics from [Dataproc](https://cloud.google.com/dataproc/) in Google Cloud Platform. It contains all metrics exported from the [GCP Dataproc Monitoring API](https://cloud.google.com/monitoring/api/metrics_gcp#gcp-dataproc).

You can specify a single region to fetch metrics like `us-central1`. Be aware that GCP Storage does not use zones so `us-central1-a` will return nothing. If no region is specified, it will return metrics from all buckets.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this reference to GCP storage appropriate here?

@@ -4,6 +4,9 @@
- description: Remove fields duplicated in ECS fields
type: enhancement
link: https://github.com/elastic/integrations/pull/3609
- description: Add GCP Dataproc Data stream
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needs update to latest gcp package version.


The `dataproc` dataset fetches metrics from [Dataproc](https://cloud.google.com/dataproc/) in Google Cloud Platform. It contains all metrics exported from the [GCP Dataproc Monitoring API](https://cloud.google.com/monitoring/api/metrics_gcp#gcp-dataproc).

You can specify a single region to fetch metrics like `us-central1`. Be aware that GCP Dataproc does not use zones so `us-central1-a` will return nothing. If no region is specified, it will return metrics from all buckets.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After reading this line, I was wondering if we can simply remove the zone configuration from the data stream. what do you think?

Suggested change
You can specify a single region to fetch metrics like `us-central1`. Be aware that GCP Dataproc does not use zones so `us-central1-a` will return nothing. If no region is specified, it will return metrics from all buckets.
You can specify a single region to fetch metrics like `us-central1`. Be aware that GCP Dataproc is a regional service. If no region is specified, it will return metrics from all buckets.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Integration:gcp Google Cloud Platform Team:Cloud-Monitoring Label for the Cloud Monitoring team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Metricbeat] Add dataproc metricset to Google Cloud module
3 participants