TechEvent Time Seriesd Databases
0 likes37 views
Time series databases are optimized for storing time-stamped sensor or event data in regular intervals, with little change to stored values. They differ from relational databases by using column-oriented storage, indexing time as a key, and supporting high write volumes, automatic data compaction, and continuous queries over aggregated time series data. Examples include InfluxDB, which uses a time-structured merge tree to optimize storage of billions of data points within a low disk footprint. While time series databases excel at certain use cases, relational databases may still be better for other types of complex, non-time-based data.
TechEvent Time Seriesd Databases
- 1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH Time Series Databases Mischa Kölliker
- 2. Time Series Databases Time Series Databases2 15.09.2018 1. The Use Case for Time Series Databases 2. What Do They Different Compared to Relational Databases? 3. Storage Structures 4. Related Stuff
- 3. Time Series Databases3 15.09.2018 The Use Case For TimeSeries DBs
- 4. Time Series Time Series Databases4 15.09.2018 Structured data, that is stored in more or less regular time intervals – Split up in simple values, that independently make sense Usually, data is not modified after initial storage (except for consolidation) Is a log file time series data?
- 5. The Use Case for Time Series DBs Time Series Databases5 15.09.2018
- 6. The Use Case for Time Series DBs Time Series Databases6 15.09.2018
- 7. Sum up – What Makes Time Series Data Special? Time Series Databases7 15.09.2018 Structured data No dependencies (FKs, JOINs) Regular time intervals Stored data does not change Values between "rows" differ only a little Old details are boring
- 8. Time Series Databases8 15.09.2018
- 9. Time Series Databases9 15.09.2018 Difference to Relational Databases
- 10. Why not use a Relational Database? Time Series Databases10 15.09.2018 Billions of individual data points Append-only or append-mostly data Often data is stored in fixed intervals Often small or no value changes between subsequent data points Less need to index values Less need for joins Bulk deletes Bulk consolidation Aggregations over time
- 11. What do Time Series DBs Differently? Time Series Databases11 15.09.2018 Column oriented vs. row oriented – Things like select count(*) is like a count over several tables in a relational DB Time(stamp) is always part of the key Differentiation between "key" fields and "value" fields – "key" fields are indexed, and have a low ordinality – "value" fields are often not indexed Periodic compaction Retention policies (automatic deletion of old data) Continuous queries
- 12. Some Limitations of Some Time Series DBs Time Series Databases12 15.09.2018 Store only numeric values (e.g. no strings) Store values at fixed time intervals only No real query language Limited network protocols support Lack of Continuous Query / Rollups / Downsampling No security No clustering support (my favourite, InfluxDB, doesn't suffer from these, except for commercial-only clustering support) see https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCEWdPtAoccJ4a-IuZv4fXDHxM/pubhtml#
- 13. time and Time literals are first class citizens Time Series Databases13 15.09.2018
- 14. Fill missing values Time Series Databases14 15.09.2018
- 15. Time Series Databases15 15.09.2018 Storage Structures
- 16. TSM – Time-Structured-Merge Trees Time Series Databases16 15.09.2018 Data is grouped into shards (each covering one week's data by default) Inverted Index https://www.slideshare.net/influxdata/inside-the-influx-db-storage-engine week – 4 (1 file) week - 3 (1 file) week - 2 (1 file) week - 1 (1 file) current week compacted, cold for write WAL "write ahead log" (append-only store)
- 17. Space-Optimized Storage (InfluxDB example) Time Series Databases17 15.09.2018 Total 40'220'369 data points 94 MB disk space over 22 shards (22 weeks) => 94*1024*1024/40220369 = 2.45 bytes per data point (including structures, indices, and so on)
- 18. After ~520 hours runtime (the java procs about 480h) Grafana runs as a Docker container InfluxDB runs standalone (i.e. is not Dockerized) Influxd – CPU and Mem usage on a RaspberryPi Time Series Databases18 15.09.2018
- 19. Time Series Databases19 15.09.2018 Related
- 20. Google Trends (1 year) Time Series Databases20 15.09.2018
- 21. …vs. InfluxDB Marketing's View Time Series Databases21 15.09.2018
- 22. Other TSDBs Time Series Databases22 15.09.2018 For a good overview and comparison, see: https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCEWdPtAoccJ4a-IuZv4fXDHxM/pubhtml# (from DalmatinerDB supporters, so quite biased)
- 23. When to TSDB, and When Not Time Series Databases23 15.09.2018 Know the use case TSDBs are not a hammer, and not all data is a nail Most reasonable systems require some additional storage Don't use it for: – Not time based data – Configuration- and master data – Complex structures – Joins – If you can't draw a graph from it, it's probably not time series data
- 24. Time Series Databases24 15.09.2018 Sum up
- 25. Sum up Time Series Databases25 15.09.2018 Use the right tool for each task If storage space can become an issue, look for optimized solutions – The same applies for performance TSDBs are not that mature yet (compared to relational databases)
- 26. Questions or Comments? Mischa Kölliker Principal Consultant Trivadis AG 15.09.2018 Time Series Databases26