Adding cloudwatch as destination for prometheus

[Paramadon]

Description of the issue

Currently Prometheus does not support Cloudwatch as a destination, users can only use Cloudwatch Logs or AMP in order to get Prometheus metrics. This pr adds the ability to have Cloudwatch be a destination for Prometheus metrics.

Description of changes

Adding Cloudwatch destination for Prometheus metrics.

Verified metrics pop up on cloudwatch:

Test setup:

agent config:

{
  "agent": {
    "metrics_collection_interval": 15,
    "run_as_user": "root",
    "debug": true
  },
  "metrics": {
    "metrics_destinations": {
            "cloudwatch":{}
    },
    "metrics_collected": {
      "prometheus": {
        "prometheus_config_path": "/home/ec2-user/prometheus/prometheus.yaml"
      }
    },
    "append_dimensions": {
      "ImageId": "${aws:ImageId}",
      "InstanceId": "${aws:InstanceId}",
      "InstanceType": "${aws:InstanceType}",
      "AutoScalingGroupName": "${aws:AutoScalingGroupName}"
    },
    "aggregation_dimensions": [
      [
        "InstanceId",
        "InstanceType"
      ]
    ]
  }
}

Prometheus yaml:

global:
  scrape_interval: 5s
  scrape_timeout: 5s
scrape_configs:
  - job_name: 'prometheus_test_job'
    static_configs:
      - targets: ['localhost:8101']

Prometheus metrics file:

prometheus_test_untyped{include="yes",prom_type="untyped"} 1
# TYPE prometheus_test_counter counter
prometheus_test_counter{include="yes",prom_type="counter"} 1
# TYPE prometheus_test_counter_exclude counter
prometheus_test_counter_exclude{include="no",prom_type="counter"} 1
# TYPE prometheus_test_gauge gauge
prometheus_test_gauge{include="yes",prom_type="gauge"} 500
# TYPE prometheus_test_histogram histogram
prometheus_test_histogram_sum{include="yes",prom_type="histogram"} 300
prometheus_test_histogram_count{include="yes",prom_type="histogram"} 75
prometheus_test_histogram_bucket{include="yes",le="0",prom_type="histogram"} 1
prometheus_test_histogram_bucket{include="yes",le="0.5",prom_type="histogram"} 2
prometheus_test_histogram_bucket{include="yes",le="2.5",prom_type="histogram"} 3
prometheus_test_histogram_bucket{include="yes",le="5",prom_type="histogram"} 4
prometheus_test_histogram_bucket{include="yes",le="+Inf",prom_type="histogram"} 5

Starting prometheus server:
sudo python3 -m http.server 8101 --directory /home/ec2-user/prometheus &> /dev/null &

[jefchien]

There's a function in common that will get the destinations for you. It's currently used in the jmx pipeline.

amazon-cloudwatch-agent/translator/translate/otel/pipeline/jmx/translators.go

Line 15 in de987cd

destinations := common.GetMetricsDestinations(conf)

[jefchien]

nit: Makes it seem like the telegraf prometheus plugin can send to CW/PMD and CWL/PLE when that is not the case.

[dricross]

Is the intention to default to CloudWatch (PMD) if no destinations are set? If so, I think this should be prometheus/cloudwatch, right?

[dricross]

left over debug logs?