OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Khavronenko - VictoriaMetrics.pdf
1 like50 views
The document discusses the specifics of data analysis in time series databases, highlighting the characteristics and applications of time series data. It emphasizes the importance of efficient data ingestion and specialized query languages for time series analysis, particularly in environments utilizing microservices and Kubernetes. Additionally, it introduces various tools and functionalities offered by VictoriaMetrics for handling time series data effectively.
OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Khavronenko - VictoriaMetrics.pdf
- 1. © 2022 VictoriaMetrics Specifics of data analysis in Time Series Databases
- 2. Co-founder of VictoriaMetrics Software engineer with experience in distributed systems, monitoring and high-performance services. https://github.com/hagen1778 https://twitter.com/hagen1778 Roman Khavronenko
- 3. The High Performance Open Source Time Series Database & Monitoring Solution 70M+ downloads 7.4k starts 180+ contributors Grammarly, CERN, StitchFix, Wix, Ably, Brandwatch, Semrush
- 4. What is time series data?
- 5. It is an observation! up 1 blood_pressure_syst 120 blood_pressure_diast 80 How are you today?
- 6. What is time series data up 1 blood_pressure_syst 120 blood_pressure_diast 80 Mon up 1 blood_pressure_syst 135 blood_pressure_diast 82 Tue up 1 blood_pressure_syst 140 blood_pressure_diast 90 Wed up 1 blood_pressure_syst 180 blood_pressure_diast 99 Thu up 0 blood_pressure_syst 0 blood_pressure_diast 0 Fri
- 7. Time Series Data Mon, Tue, Wed… 120 => 135 => 140… blood_pressure_syst 120 What is time series data
- 8. Where to use time series data?
- 9. The SRE Report 2023
- 10. Kubernetes!
- 11. What kubernetes changed? ● Split application into microservices
- 12. What kubernetes changed? ● Split application into microservices ● Deploy multiple replicas of each microservice
- 13. What kubernetes changed? ● Split application into microservices ● Deploy multiple replicas of each microservice ● Each cluster, instance, pod, container exposes metrics
- 14. What kubernetes changed? ● Split application into microservices ● Deploy multiple replicas of each microservice ● Each cluster, instance, pod, container exposes metrics ● HPA, Preemptible nodes
- 15. What kubernetes changed? ● Split application into microservices ● Deploy multiple replicas of each microservice ● Each cluster, instance, pod, container exposes metrics ● HPA, Preemptible nodes ● Pods average lifespan is a couple of days
- 17. What is special about time series data? ● <value> is always associated with <timestamp> ● Series are continuous ● Time series data is unlikely to be changed in past
- 18. What is special about time series data? <value> is always associated with <timestamp> ● Data can be structured in tuples of <timestamp:value> pairs ● Read API should expect <timestamp> filter ● <values> are always sorted by <timestamp>
- 19. What is special about time series data? Series are continuous: ● Appends are more likely than registering of the new series ● Compression for sorted list of <timestamps> and <values> can be very efficient ● Each data tuple is connected with previous one, so you plot lines not dots
- 20. What is special about time series data? Time series data is unlikely to be changed in past: ● UPDATE/UPSERT/DELETE are not important ● Focus on ingestion and reading
- 21. TimeSeries specialized solution focuses on ● Providing max write throughput ● Prioritize writes over reads ● Use optimal data layout on disk ● Provide best compression based on data specifics ● Support optimal query language for time series
- 22. Why not just use SQL?
- 24. Prometheus Query Language query language for selecting and aggregating time series data in real time
- 27. CREATE TABLE CREATE TABLE electricity ( datetime DateTime, address String, value Float64 ) INSERT INTO electricity VALUES ('2022-10-01 00:00:00', 'Sheen Court 150', 0) ('2022-10-01 00:00:00', 'Richmond Hill 2', 0) ('2022-10-01 00:00:00', 'Belvedere Road 1', 0) …
- 29. SQL=> $columns(address, sum(value) c) FROM electricity
- 42. Ephemeral data point always repeats the left closest real point
- 45. Counter reset detection Point N Point N+1 Point N+1 < Point N ?
- 49. Speed (rate) of the metric change
- 50. Speed (rate) of the metric change
- 51. Node Exporter: most popular Grafana dashboard
- 52. Node Exporter: most popular Grafana dashboard Has 20 Mil downloads and 60 reviews! Contains 230 PromQL queries in it: ● ~120 queries are selectors, e.g. node_textfile_scrape_total; ● ~80 queries calculate rate, e.g. rate(node_textfile_scrape_total); ● the rest are aggregation functions like sum or count. It is easier than you think!
- 53. Time Series analysis specifics ● Queries need to be simple to write and understand ● It is natural to have time params in Query API ● Query step helps to align time series resolution ● Counter reset detection makes life easier
- 54. ● Simpler to use query language MetricsQL ● More than 100 additional functions in MetricsQL ● Query API enhancements ● Automatic step and interval detection ● Anomaly detection ● And many more! Looking for more? Check VictoriaMetrics!
- 55. Questions? Links: ● https://github.com/VictoriaMetrics/VictoriaMetrics ● https://www.youtube.com/watch?v=oJ-RIkBWzAM ● The SRE Report 2023 ● ESG Research Report ● VictoriaMetrics blog