Pipeline all the Dashboards as Code
0 likes•667 views
Kris Buytaert discusses the problems with dashboard sprawl in Grafana and proposes managing dashboards as code. He introduces Dashboard Manager, a tool for fetching dashboards from a development Grafana instance, comparing them, and uploading them to a production instance. This allows end users to create dashboards in development and then promote them to production in a version controlled and reproducible manner.
Pipeline all the Dashboards as Code
- 1. OSMC Kris Buytaert @krisbuytaert Pipeline all the Dashboards as Code ? November , 2021
- 2. • I used to be a developer • Then I became an Ops person • Chief Trolling/Travel/Technical Officer @ Inuits.eu • Everything is a Freaking DNS Problem • DNS : devops needs sushi • Buiding Clouds since before the Bookstore • Organiser of #devopsdays, #cfgmgmtcamp, #loadays, ... Who am I ? @krisbuytaert
- 3. • Operational Metrics • Application Metrics • Business Metrics • Travel time to home during Traffic Jams • Time between last conference • Legobricks ordered • Sandwiches ordered by the team Dashboard all the things @krisbuytaert
- 4. • People edit a Dashboard • People clone a Dashboard • People create a new Dashboard • People "improve" other people's Dashboards • People delete other people's Dashboards class {'chaos': ensure => present } Dashboards Sprawl @krisbuytaert
- 5. • No more hacking in production No more manual changes? @krisbuytaert
- 6. What do we want @krisbuytaert
- 7. Jsonnet is a DSL to build JSON Data, JSON On steroids Jsonnet is dead Jsonnet is alive Grafonet GrafanaLib Dashboard as Code @krisbuytaert
- 8. Look a Squirrel ! @krisbuytaert
- 9. We are all JSON engineers We are all YAML engineers The average end user who wants to create his own dashboard is not a devoops YAML Engineer This is just too complex Problems with Dashboard as Code @krisbuytaert
- 10. • A dev Grafana • Dashboard as Code • Dashboard for Dirty Clickers • A prod Grafana (read only) • A way to promote from dev to prod What we really need @krisbuytaert
- 11. https://github.com/inuits/dashboard-manager A command line utility to manage and promote dashboards between Grafana instances, coupled with gitlab-ci. Dashboard Manager for Grafana @krisbuytaert
- 12. grafana_instances_output: - api_key_file: production-secret url: http://127.0.0.1:3000 name: prod grafana_instances_input: - api_key_file: dev-secret url: https://remote-dev.example.com/ name: dev http_client: # Configured like alertmanager http client: # https://prometheus.io/docs/alerting/latest/configuration/#http_conf tls_config: insecure_skip_verify: true Example Configuration @krisbuytaert
- 13. fetch --output-directory=OUTPUT-DIRECTORY Fetch dashboards from input grafana. compare --dashboards-directory=DASHBOARDS-DIRECTORY --results=RESULTS Compare dashboards. upload --dashboards-directory=DASHBOARDS-DIRECTORY --input-instance=INPUT-I --dashboards=DASHBOARDS Upload dashboards. Usage @krisbuytaert
- 14. Step 1 : User creates Dashboard in dev @krisbuytaert
- 15. Step 2 : Trigger the pipeline @krisbuytaert
- 16. Step 3 : Check downstream Created Jobs @krisbuytaert
- 17. Step 4 : Promote the (individual) Dashboard @krisbuytaert
- 18. Step 5 : New dashboard is in prod @krisbuytaert
- 19. • Usable for End Users • Version controlled • Reproducable • Preventing dashboard Sprawl • Can still be used for dashboards from "real" code. to do the promotions Conclusion @krisbuytaert
- 20. Kris Buytaert @krisbuytaert [email protected] Essensteenweg 31 2930 Brasschaat Belgium Contact: [email protected] +32-3-8082105