Getting Started: Intro to Telegraf - July 2021
1 like•620 views
The document is a guide on Telegraf, an agent for collecting and reporting metrics and events for the InfluxData platform. It details the plugin ecosystem, configuration basics, and how to extend it, highlighting its low-code, customizable nature and strong community support. Telegraf facilitates efficient data acquisition through various methods, plugins, and transformation capabilities, making it a versatile tool for managing metrics and logs.
![© 2021 InfluxData. All rights reserved.
37
37
[[processors.execd]]
command = ["python", "./processors/forecasting.py"]
Sample ExecD Config: Input, Output, Processor](https://image.slidesharecdn.com/intrototelegraf-july82021-210707233750/85/Getting-Started-Intro-to-Telegraf-July-2021-37-320.jpg)
Getting Started: Intro to Telegraf - July 2021
- 1. Samantha Wang Product Manager, InfluxData Getting Started: Intro to Telegraf
- 2. © 2021 InfluxData. All rights reserved. 2 2 – What is Telegraf – Plugin Ecosystem – Configuration Basics – Extending the Ecosystem Agenda
- 3. © 2021 InfluxData. All rights reserved. 3 InfluxDB
- 4. © 2021 InfluxData. All rights reserved. 4 What is InfluxDB? Characteristics of the data – Time-stamped – Generated in regular (metric) and irregular (event) time periods – Huge volumes – Real time and time sensitive
- 5. © 2021 InfluxData. All rights reserved. 5 Visualization Alerts Triggers Metrics Logs Traces Events The platform of choice for all metrics & event workloads
- 6. © 2021 InfluxData. All rights reserved. 6 InfluxData Platform Architecture
- 7. © 2021 InfluxData. All rights reserved. 7 InfluxData products and offerings * Availability TBD
- 8. © 2021 InfluxData. All rights reserved. 8 Telegraf
- 9. © 2021 InfluxData. All rights reserved. 9 9 Current Data Acquisition - All Methods Agent-based Push (aka Telegraf) Agentless Pull (aka Scrapers) Client Libraries Native/Ecosystem ● 200+ Telegraf plugins ● Regular cadence of releases ● Why use it? ○ No code ○ Large community ○ Lightweight but powerful ○ Customizable ● Prometheus scraper (OSS only) ● Flux prometheus.from ● Flux csv.from(url) ● Why use them? ○ Get data in quickly ○ Doesn’t require agent downloads on monitoring device ● 7 Libraries: Python, C#, Java, GO, Javascript/Node.js, Ruby, PHP ● Handles batching, chunking, setting right headers, etc. ● Why use them? ○ Easy way to get started ○ Need libraries when building custom applications ● Source system speaks line protocol ● Examples: JMeter, NiFi, Vector, FluentD ● Influx CLI CSV Import ● Why use them? ○ Know what you want to monitor, quick and easy integration
- 10. © 2021 InfluxData. All rights reserved. 10 Telegraf provides the benefits of… ● Low/No code ● Robust scheduler ● High-speed ingestion ● Full-streaming support ● Metric routing ● Flexible parsing, formatting, serializing ● Customizable and Extensible (ExecD plugins, Starlark) Instead of… ● Writing long data scraping scripts ● Worrying about unreliable data collection ● Trouble scaling your data collection ● Resulting in messy data ● Having a lot of unnecessary data in your database
- 11. © 2021 InfluxData. All rights reserved. 11 11 Telegraf: Agent for Collecting Metrics & Events Plugin-driven server agent for collecting and reporting metrics • Written in Go • Single Binary, No external dependencies • Minimal memory footprint • Optimized for writing to InfluxDB • Optimized for streaming data Input plugins to source a variety of metrics • directly from the system it’s running on • pull metrics from third party APIs • listen for metrics via a statsd and Kafka consumer services
- 12. © 2021 InfluxData. All rights reserved. 12 12 Telegraf: Agent for Collecting Metrics & Events Output plugins to send metrics to a variety of other datastores, services, and message queues • InfluxDB, Graphite, OpenTSDB, Datadog, and many others • Writes to multiple outputs at the same time Metrics routing and conversion Aggregation stage for InfluxDB
- 13. © 2021 InfluxData. All rights reserved. 13 13 Telegraf Agent – Core of Telegraf functionality providing the following – Robust scheduler – Adjustments for clock-drift – Adjustments for job scheduling issues that may occur – In-memory metric buffers – Metric tracking with flow back-pressure in plugins like Kafka – Full-streaming support – Metric routing: name & field pass & drop – Flexible parsing, formatting, serializing
- 14. © 2021 InfluxData. All rights reserved. 14 Plugins! Plugins! Plugins!
- 15. © 2021 InfluxData. All rights reserved. 15 15 Telegraf Plugin Ecosystem Input Plugins ● collect metrics from system, services, or third party APIs Output Plugins ● write data to various destinations Processors ● transform, decorate, and/or filter metrics Aggregators ● create aggregate metrics (e.g. mean, min, max, quantiles, etc.)
- 16. © 2021 InfluxData. All rights reserved. 16 16 Telegraf Plugin Types 170+ 40+ 15+ 5+ Input Output Processors Aggregators
- 17. © 2021 InfluxData. All rights reserved. 17 17 Input Plugins activemq aerospike amqp_consumer apache apcupsd aurora azure_storage_que ue bcache beanstalkd bind bond burrow cassandra ceph cgroup chrony cisco_telemetry_mdt clickhouse cloud_pubsub cloud_pubsub_push cloudwatch conntrack consul couchbase couchdb cpu dcos disk diskio disque dmcache dns_query docker docker_log dovecot ecs elasticsearch ethtool eventhub_consumer exec execd fail2ban fibaro file filecount filestat fireboard fluentd github gnmi graylog haproxy hddtemp http http_listener_v2 http_response httpjson icinga2 infiniband influxdb
- 18. © 2021 InfluxData. All rights reserved. 18 18 Input Plugins (2) influxdb_listener influxdb_v2_listener intel_rdt internal interrupts ipmi_sensor ipset iptables ipvs jenkins jolokia jolokia2 jti_openconfig_telemetr y kafka_consumer kafka_consumer_legacy kapacitor kernel kernel_vmstat kibana kinesis_consumer kube_inventory kubernetes lanz leofs linux_sysctl_fs logparser logstash lustre2 mailchimp marklogic mcrouter mem memcached mesos minecraft modbus mongodb monit mqtt_consumer multifile mysql nats nats_consumer neptune_apex net net_response nginx nginx_plus nginx_plus_api nginx_sts nginx_upstream_check nginx_vts nsd nsq nsq_consumer nstat ntpq nvidia_smi opcua openldap
- 19. © 2021 InfluxData. All rights reserved. 19 19 Input Plugins (3) openntpd opensmtpd openweathermap passenger pf pgbouncer phpfpm ping postfix postgresql postgresql_extensibl e powerdns powerdns_recursor processes procstat prometheus proxmox puppetagent rabbitmq raindrops ras redfish redis rethinkdb riak salesforce sensors sflow smart snmp snmp_legacy snmp_trap socket_listener solr sqlserver stackdriver statsd suricata swap synproxy syslog sysstat system systemd_units tail tcp_listener teamspeak temp tengine tomcat trig twemproxy udp_listener unbound uwsgi varnish vsphere webhooks win_eventlog win_perf_counte rs win_services wireguard wireless x509_cert zfs zipkin zookeeper
- 20. © 2021 InfluxData. All rights reserved. 20 20 Processor Plugins clone converter date dedup defaults enum execd filepath ifname override parser pivot port_name printer regex rename reverse_dns s2geo starlark strings tag_limit template topk unpivot
- 21. © 2021 InfluxData. All rights reserved. 21 21 Aggregator Plugins basicstats final histogram merge minmax valuecounter
- 22. © 2021 InfluxData. All rights reserved. 22 22 Output Plugins amon amqp application_insights azure_monitor cloud_pubsub cloudwatch cratedb datadog discard dynatrace elasticsearch exec execd file graphite graylog health http influxdb influxdb_v2 instrumental kafka kinesis librato logzio mqtt nats newrelic nsq opentsdb prometheus_client riemann riemann_legacy socket_writer stackdriver sumologic syslog timestream warp10 wavefront yandex_cloud_monito ring
- 23. © 2021 InfluxData. All rights reserved. 23 23 Telegraf CPU Mem Disk Docker Kubernetes /metrics Kafka MySQL Process - transform - decorate - filter Aggregate - mean - min,max - count - variance - stddev InfluxDB File Kafka CloudWatch CloudWatch Input Output One Telegraf, Multiple Plugins
- 24. © 2021 InfluxData. All rights reserved. 24 24 Can’t find the plugin you need? Telegraf is 100% open source with a strong community of contributors It’s easy to write your own Telegraf plugin! 1) Follow the contribution guide for Go 2) Write your plugin in any language and run it externally with ExecD
- 25. © 2021 InfluxData. All rights reserved. 25 25 Popular Plugins Out-of-the-box Custom Kubernetes (kubelet) HTTP/socket listener Kube_inventory (apiserver) HTTP (formatted endpoints) Kafka (consumer) Prometheus (/metrics) SNMP Exec/Execd AMQP (mq metadata) StatsD Redis Nginx HAproxy Jolokia2
- 26. © 2021 InfluxData. All rights reserved. 26 26 Parsing JSON CSV Graphite Prometheus CollectD Dropwizard Form URL-encoded Grok
- 27. © 2021 InfluxData. All rights reserved. 27 Getting Started with Telegraf
- 28. © 2021 InfluxData. All rights reserved. 28 28 Download Telegraf
- 29. © 2021 InfluxData. All rights reserved. 29 Configuration Basics
- 30. © 2021 InfluxData. All rights reserved. 30 30 Telegraf Configuration Snippets Include many plugins per Telegraf - multiple inputs, outputs, processors, aggregators
- 31. © 2021 InfluxData. All rights reserved. 31 31 The Telegraf config file needs to be specified for Telegraf agent to operate properly. It contains setup for the agent, global tags, and enabled outputs (through commenting out or removing unnecessary lines) What’s in a configuration file?
- 32. © 2021 InfluxData. All rights reserved. 32 32 Set and Test Your Configuration $ telegraf --input-filter cpu:system --output-filter influxdb --processor-filter starlark --aggregator-filter valuecounter config > telegraf.conf WRITE DESIRED CONFIG FILE RUN CONFIG FILE TEST $ telegraf --config telegraf.conf --test VERIFY YOUR CONFIG WORKS
- 33. © 2021 InfluxData. All rights reserved. 33 Extending the Ecosystem
- 34. © 2021 InfluxData. All rights reserved. 34 34 ● Work with the open-source community or submit upgrades or enhancements for existing plugins ● Use ExecD to write a plugin in Go or a language of your choice You can extend Telegraf by:
- 35. © 2021 InfluxData. All rights reserved. 35 35 ● Plugin runs in its own process ● Requires line protocol ● Avoid the need for review by Telegraf team ● Supports the same API as an internal plugin ● Can use for non-GO plugins ● Can use for licensed software plugins ● Can use for any type of plugin (input, output, processor, aggregator) External plugins via ExecD
- 36. © 2021 InfluxData. All rights reserved. 36 36 Execd Plugin Arch.
- 37. © 2021 InfluxData. All rights reserved. 37 37 [[processors.execd]] command = ["python", "./processors/forecasting.py"] Sample ExecD Config: Input, Output, Processor
- 38. © 2021 InfluxData. All rights reserved. 38 38 Customer Quotes “Telegraf is like a swiss army knife for connecting various MQTT sources and OPC UA sources.” — Fr. Ant. Niedermayr “Our next-generation pipeline takes advantage of Kafka and the Telegraf streaming service to create a more robust data topology. Essentially this allows us to explicitly implement the four R’s: routability, retention, resilience, and redundancy.” — Wayfair
- 39. © 2021 InfluxData. All rights reserved. 39 Telegraf Community
- 40. © 2021 InfluxData. All rights reserved. 40 Get involved with Telegraf • Telegraf GitHub: github.com/influxdata/telegraf • Community Slack: influxdata.com/slack #telegraf #telegraf-dev • Community Website: community.influxdata.com
- 41. © 2021 InfluxData. All rights reserved. 41 41 References Getting started with Telegraf Telegraf plugins Telegraf GitHub Page External Plugins Guide
- 42. Thank You
- 43. © 2021 InfluxData. All rights reserved. 43 43 Input Plugins activemq aerospike amqp_consumer apache apcupsd aurora azure_storage_queu e bcache beanstalkd bind bond burrow cassandra ceph cgroup chrony cisco_telemetry_mdt clickhouse cloud_pubsub cloud_pubsub_push cloudwatch conntrack consul couchbase couchdb cpu dcos disk diskio disque dmcache dns_query docker docker_log dovecot ecs elasticsearch ethtool eventhub_consumer exec execd fail2ban fibaro file filecount filestat fireboard fluentd github gnmi graylog haproxy hddtemp http http_listener_v2 http_response httpjson icinga2 infiniband influxdb
- 44. © 2021 InfluxData. All rights reserved. 44 44 Input Plugins (2) influxdb_listener influxdb_v2_listener intel_rdt internal interrupts ipmi_sensor ipset iptables ipvs jenkins jolokia jolokia2 jti_openconfig_telemetry kafka_consumer kafka_consumer_legacy kapacitor kernel kernel_vmstat kibana kinesis_consumer kube_inventory kubernetes lanz leofs linux_sysctl_fs logparser logstash lustre2 mailchimp marklogic mcrouter mem memcached mesos minecraft modbus mongodb monit mqtt_consumer multifile mysql nats nats_consumer neptune_apex net net_response nginx nginx_plus nginx_plus_api nginx_sts nginx_upstream_check nginx_vts nsd nsq nsq_consumer nstat ntpq nvidia_smi opcua openldap
- 45. © 2021 InfluxData. All rights reserved. 45 45 Input Plugins (3) openntpd opensmtpd openweathermap passenger pf pgbouncer phpfpm ping postfix postgresql postgresql_extensibl e powerdns powerdns_recursor processes procstat prometheus proxmox puppetagent rabbitmq raindrops ras redfish redis rethinkdb riak salesforce sensors sflow smart snmp snmp_legacy snmp_trap socket_listener solr sqlserver stackdriver statsd suricata swap synproxy syslog sysstat system systemd_units tail tcp_listener teamspeak temp tengine tomcat trig twemproxy udp_listener unbound uwsgi varnish vsphere webhooks win_eventlog win_perf_counte rs win_services wireguard wireless x509_cert zfs zipkin zookeeper
- 46. © 2021 InfluxData. All rights reserved. 46 46 Processor Plugins clone converter date dedup defaults enum execd filepath ifname override parser pivot port_name printer regex rename reverse_dns s2geo starlark strings tag_limit template topk unpivot
- 47. © 2021 InfluxData. All rights reserved. 47 47 Aggregator Plugins basicstats final histogram merge minmax valuecounter
- 48. © 2021 InfluxData. All rights reserved. 48 48 Output Plugins amon amqp application_insights azure_monitor cloud_pubsub cloudwatch cratedb datadog discard dynatrace elasticsearch exec execd file graphite graylog health http influxdb influxdb_v2 instrumental kafka kinesis librato logzio mqtt nats newrelic nsq opentsdb prometheus_client riemann riemann_legacy socket_writer stackdriver sumologic syslog timestream warp10 wavefront yandex_cloud_monitor ing