collectd & PostgreSQL
8 likes13,787 views
This document details the author's experience implementing collectd in a high-performance PostgreSQL environment, highlighting features of collectd and challenges faced with data monitoring and visualization. After facing limitations with various tools and needing efficient data handling for over a million database objects, the author demonstrates how to configure collectd, develop custom queries, and optimize data collection and storage. The presentation also outlines hurdles encountered and solutions devised for effective data management and visualization, concluding with acknowledgments and licensing details.
![Random: What the Write HTTP JSON data looks like
Note: Write HTTP packs as much data as it can into a 4KB buļ¬er.
[ {
"values": [197141504, 175136768],
"dstypes": ["counter", "counter"],
"dsnames": ["read", "write"],
"time": 1251533299,
"interval": 10,
"host": "leeloo.lan.home.verplant.org",
"plugin": "disk",
"plugin_instance": "sda",
"type": "disk_octets",
"type_instance": ""
}, ... ]
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 30 / 43](https://image.slidesharecdn.com/pgcollectd-111118090627-phpapp01/85/collectd-PostgreSQL-30-320.jpg)
![Database design
Table "collectd.value_list"
Column | Type | Modifiers
-----------------+--------------------------+-----------
time | timestamp with time zone | not null
interval | integer | not null
host | character varying(64) | not null
plugin | character varying(64) | not null
plugin_instance | character varying(64) |
type | character varying(64) | not null
type_instance | character varying(64) |
dsnames | character varying(512)[] | not null
dstypes | character varying(8)[] | not null
values | numeric[] | not null
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 33 / 43](https://image.slidesharecdn.com/pgcollectd-111118090627-phpapp01/85/collectd-PostgreSQL-33-320.jpg)
collectd & PostgreSQL
- 1. collectd & PostgreSQL Mark Wong [email protected] [email protected] PDXPUG November 17, 2011
- 2. My Story ā¢ How did I get to collectd? ā¢ What is collectd ā¢ Hacking collectd ā¢ Using collectd with Postgres ā¢ Visualizing the data markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 2 / 43
- 3. Brief background ā¢ Working at a little company called Emma http://myemma.com ā¢ Collect performance data from production systems markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 3 / 43
- 4. What did we have? ā¢ A database with over 1 million database objects ā¢ >500,000 tables ā¢ >1,000,000 indexes ā¢ Tables alone generate 11,000,000 data point per sample markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 4 / 43
- 5. What did we try? Only free things: ā¢ Cacti http://www.cacti.net/ ā¢ Ganglia http://ganglia.info/ ā¢ Munin http://munin-monitoring.org/ ā¢ Reconnoiter https://labs.omniti.com/labs/reconnoiter ā¢ Zenoss http://community.zenoss.org/ markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 5 / 43
- 6. What doesnāt work Dependency on RRDtool; canāt handle more than hundreds of thousands of metrics (Application Buļ¬er-Cache Management for Performance: Running the Worldās Largest MRTG by David Plonka, Archit Gupta and Dale Carder, LISA 2007): ā¢ Cacti ā¢ Ganglia ā¢ Munin ā¢ Reconnoiter ā¢ Zenoss markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 6 / 43
- 7. Reconnoiter almost worked for us Proās: ā¢ Write your own SQL queries to collect data from Postgres ā¢ Used Postgres instead of RRDtool for storing data ā¢ JavaScript based on-the-ļ¬y charting ā¢ Support for integrating many other health and stats collection solutions Conās: ā¢ Data collection still couldnāt keep up; maybe needed more tuning ā¢ Faster hardware? (using VMās) ā¢ More hardware? (scale out MQ processes) markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 7 / 43
- 8. Couldnāt bring myself to try anything else ā¢ Hands were tied, no resources available to help move forward. ā¢ Can we build something light weight? ā¢ Played with collectd (http://collectd.org/) while evaluating Reconnoiter markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 8 / 43
- 9. What is collectd? collectd is a daemon which collects system performance statistics periodically and provides mechanisms to store the values in a variety of ways, for example in RRD ļ¬les. http://collectd.org/ markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 9 / 43
- 10. Does this look familiar? Note: RRDtool is an option, not a requirement markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 10 / 43
- 11. What is special about collectd? From their web site: ā¢ itās written in C for performance and portability ā¢ includes optimizations and features to handle hundreds of thousands of data sets ā¢ PostgreSQL plugin enables querying the database ā¢ Can collect most operating systems statistics (I say āmostā because I donāt know if anything is missing) ā¢ Over 90 total plugins http://collectd.org/wiki/index.php/Table_of_Plugins markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 11 / 43
- 12. collectd data description ā¢ time - when the data was collected ā¢ interval - frequency of data collection ā¢ host - server hostname ā¢ plugin - collectd plugin used ā¢ plugin instance - additional plugin information ā¢ type - type of data collected for set of values ā¢ type instance - unique identiļ¬er of the metric ā¢ dsnames - names for the values collected ā¢ dstypes - type of data for values collected (e.g. counter, gauge, etc.) ā¢ values - array of values collected markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 12 / 43
- 13. PostgreSQL plugin conļ¬guration Deļ¬ne custom queries in collectd.conf: LoadPlugin postgresql <Plugin postgresql> <Query magic> Statement "SELECT magic FROM wizard;" <Result> Type gauge InstancePrefix "magic" ValuesFrom magic </Result> </Query> ... markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 13 / 43
- 14. . . . per database. ... <Database bar> Interval 60 Service "service_name" Query backend # predefined Query magic_tickets </Database> </Plugin> Full details at http://collectd.org/wiki/index.php/Plugin:PostgreSQL markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 14 / 43
- 15. Hurdles More meta data: ā¢ Need a way to save schema, table, and index names; canāt diļ¬erentiate stats between tables and indexes ā¢ Basic support of meta data in collectd but mostly unused ā¢ How to store data in something other than RRDtool markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 15 / 43
- 16. Wanted: additional meta data Hack the PostgreSQL plugin to create meta data for: ā¢ database - database name (maybe not needed, same as plugin instance) ā¢ schemaname - schema name ā¢ tablename - table name ā¢ indexname - index name ā¢ metric - e.g. blks hit, blks read, seq scan, etc. markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 16 / 43
- 17. Another database query for collecting a table statistic <Query table_stats> SELECT schemaname, relname, seq_scan FROM pg_stat_all_tables; <Query> markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 17 / 43
- 18. Identify the data <Result> Type counter InstancePrefix "seq_scan" InstancesFrom "schemaname" "relname" ValuesFrom "seq_scan" </Result> markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 18 / 43
- 19. Meta data speciļ¬c parameters <Database postgres> Host "localhost" Query table_stats SchemanameColumn 0 TablenameColumn 1 </Database> Note: The database name is set by what is speciļ¬ed in the <Database>tag, if it is not retrieved by the query. markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 19 / 43
- 20. Example data ā¢ time: 2011-10-20 18:04:17-05 ā¢ interval: 300 ā¢ host: pong.int ā¢ plugin: postgresql ā¢ plugin instance: sandbox ā¢ type: counter ā¢ type instance: seq scan-pg catalog-pg class ā¢ dsnames: {value} ā¢ dstypes: {counter} ā¢ values: {249873} markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 20 / 43
- 21. Example meta data ā¢ database: sandbox ā¢ schemaname: pg catalog ā¢ tablename: pg class ā¢ indexname: ā¢ metric: seq scan markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 21 / 43
- 22. Now what? Handās were tied (I think I mentioned that earlier); open sourced work to date: ā¢ collectd forked with patches https://github.com/mwongatemma/collectd ā¢ YAMS https://github.com/myemma/yams markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 22 / 43
- 23. Yet Another Monitoring System markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 23 / 43
- 24. Switching hats and boosting code Using extracurricular time working on equipment donated to Postgres from SUN, IBM, and HP to continue prooļ¬ng collectd changes. markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 24 / 43
- 25. How am I going to move the data? Options from available write plugins; guess which I used: ā¢ Carbon - Graphiteās storage API to Whisper http://collectd.org/wiki/index.php/Plugin:Carbon ā¢ CSV http://collectd.org/wiki/index.php/Plugin:CSV ā¢ Network - Send/Receive to other collectd daemons http://collectd.org/wiki/index.php/Plugin:Network ā¢ RRDCacheD http://collectd.org/wiki/index.php/Plugin:RRDCacheD ā¢ RRDtool http://collectd.org/wiki/index.php/Plugin:RRDtool ā¢ SysLog http://collectd.org/wiki/index.php/Plugin:SysLog ā¢ UnixSock http://collectd.org/wiki/index.php/Plugin:UnixSock ā¢ Write HTTP - PUTVAL (plain text), JSON http://collectd.org/wiki/index.php/Plugin:Write_HTTP markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 25 / 43
- 26. Process of elimination If RRDtool (wriiten in C) canāt handle massive volumes of data, a Python RRD like database probably canāt either: ā¢ Carbon ā¢ CSV ā¢ Network ā¢ RRDCacheD ā¢ RRDtool ā¢ SysLog ā¢ UnixSock ā¢ Write HTTP - PUTVAL (plain text), JSON markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 26 / 43
- 27. Process of elimination Writing to other collectd daemons or just locally doesnāt seem useful at the moment: ā¢ CSV ā¢ Network ā¢ SysLog ā¢ UnixSock ā¢ Write HTTP - PUTVAL (plain text), JSON markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 27 / 43
- 28. Process of elimination Letās try CouchDBās RESTful JSON API! ā¢ CSV ā¢ SysLog ā¢ Write HTTP - PUTVAL (plain text), JSON markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 28 / 43
- 29. Random: What Write HTTP PUTVAL data looks like Note: Each PUTVAL is a single line but is broken up into two lines to ļ¬t onto the slide. PUTVAL leeloo.lan.home.verplant.org/disk-sda/disk_octets interval=10 1251533299:197141504:175136768 PUTVAL leeloo.lan.home.verplant.org/disk-sda/disk_ops interval=10 1251533299:10765:12858 PUTVAL leeloo.lan.home.verplant.org/disk-sda/disk_time interval=10 1251533299:5:140 PUTVAL leeloo.lan.home.verplant.org/disk-sda/disk_merged interval=10 1251533299:4658:29899 markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 29 / 43
- 30. Random: What the Write HTTP JSON data looks like Note: Write HTTP packs as much data as it can into a 4KB buļ¬er. [ { "values": [197141504, 175136768], "dstypes": ["counter", "counter"], "dsnames": ["read", "write"], "time": 1251533299, "interval": 10, "host": "leeloo.lan.home.verplant.org", "plugin": "disk", "plugin_instance": "sda", "type": "disk_octets", "type_instance": "" }, ... ] markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 30 / 43
- 31. I didnāt know anything about CouchDB at the time ā¢ Query interface not really suited for retrieving data to visualize ā¢ Insert performance not suited for millions of metrics of data over short intervals (can insert same data into Postgres several orders of magnitude faster) markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 31 / 43
- 32. Now where am I going to put the data? Hoping that using the Write HTTP is still a good choice: ā¢ Write an ETL ā¢ Table partitioning logic; creation of partition tables ā¢ Transform JSON data into INSERT statements ā¢ Use Postgres markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 32 / 43
- 33. Database design Table "collectd.value_list" Column | Type | Modifiers -----------------+--------------------------+----------- time | timestamp with time zone | not null interval | integer | not null host | character varying(64) | not null plugin | character varying(64) | not null plugin_instance | character varying(64) | type | character varying(64) | not null type_instance | character varying(64) | dsnames | character varying(512)[] | not null dstypes | character varying(8)[] | not null values | numeric[] | not null markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 33 / 43
- 34. Take advantage of partitioning At least table inheritance in Postgresā case; partition data by plugin markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 34 / 43
- 35. Child table Table "collectd.vl_postgresql" Column | Type | Modifiers -----------------+--------------------------+----------- ... database | character varying(64) | not null schemaname | character varying(64) | tablename | character varying(64) | indexname | character varying(64) | metric | character varying(64) | not null Check constraints: "vl_postgresql_plugin_check" CHECK (plugin::text = āpostgresqlā::text) Inherits: value_list markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 35 / 43
- 36. How much partitioning? Lots of straightforward options: ā¢ Date ā¢ Database ā¢ Schema ā¢ Table ā¢ Index ā¢ Metric markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 36 / 43
- 37. Back to the ETL Parameters set for fastest path to working prototype: ā¢ Keeping using HTTP POST (Write HTTP plugin) for HTTP protocol and JSON ā¢ Use Python for built in HTTP Server and JSON parsing (Emma is primarily a Python shop) ā¢ Use SQLAlchemy/psycopg2 markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 37 / 43
- 38. Back again to the ETL Python didnāt perform; combination of JSON parsing, data transformation, and INSERT performance still several orders of magnitude below acceptable levels: ā¢ redis to queue data to transform ā¢ lighttpd for the HTTP interface ā¢ fastcgi C program to push things to redis ā¢ multi-threaded C program using libpq for Postgres API ā¢ pop data out of redis ā¢ table partitioning creation logic ā¢ transform JSON data into INSERT statements markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 38 / 43
- 39. Success? ā¢ Table statistics for 1 million tables collect in approximately 12 minutes. ā¢ Is that acceptable? ā¢ Can we go faster? markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 39 / 43
- 40. If you donāt have millions of data Easier ways to visualize the data: ā¢ RRDtool ā¢ RRDtool compatible front-ends http://collectd.org/wiki/index.php/List_of_front-ends ā¢ Graphite with the Carbon and Whisper combo http://graphite.wikidot.com/ ā¢ Reconnoiter markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 40 / 43
- 41. __ __ / ~~~/ . o O ( Thank you! ) ,----( oo ) / __ __/ /| ( |( ^ /___ / | |__| |__|-" markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 41 / 43
- 42. Acknowledgements Hayley Jane Wakenshaw __ __ / ~~~/ ,----( oo ) / __ __/ /| ( |( ^ /___ / | |__| |__|-" markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 42 / 43
- 43. License This work is licensed under a Creative Commons Attribution 3.0 Unported License. To view a copy of this license, (a) visit http://creativecommons.org/licenses/by/3.0/us/; or, (b) send a letter to Creative Commons, 171 2nd Street, Suite 300, San Francisco, California, 94105, USA. markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 43 / 43