What’s Evolving in the Elastic Stack
2 likes•1,441 views
The document provides an overview of the Elastic Stack, which includes components like Elasticsearch, Logstash, Beats, and Kibana. It discusses various features such as data ingestion, file integrity monitoring, data rollups, cross-cluster search, and security measures. Additionally, it covers advancements in version upgrades, including zero downtime upgrades and index lifecycle management.
What’s Evolving in the Elastic Stack
- 1. !1 What’s Evolving in the Elastic Stack
- 3. Elastic Stack Elasticsearch - search and analytics engine
- 4. Elastic Stack Beats - lightweight data shippers
- 5. Elastic Stack Logstash - ELT pipeline with multiple outputs
- 6. Elastic Stack Kibana - UI Platform
- 9. Add Data UI
- 10. Logstash - Netflow module
- 11. Logstash - Arcsight module
- 13. Auditbeat File Integrity Monitoring and Linux Kernel Auditing • Watches for file changes on Linux, macOS, Windows • Detects short lived processes and connections • Indexes directly into Elasticsearch • Correlates kernel audit events • Resolves user IDs to user names
- 14. !14 With containers architecture, everything is a moving target
- 15. !15
- 16. !16
- 18. Logstash: Persistent Queue input filter output
- 19. Logstash: Persistent Queue input Disk queue filter output
- 20. Logstash: Dead Letter Queue input Disk queue Dead letter queue filter output
- 21. Logstash: Dead Letter Queue input Disk queue Dead letter queue filter output DLQ input filter output
- 22. Disk queue PublishEvent () OutputACK ACK libbeat Beats: Spool-to-Disk
- 24. Indexing
- 25. Faster Indexing 1.x 2.x 5.x 6.x 7.x
- 26. 0 25 50 75 100 5.x 6.0 OOTB _all disabled Disk on a diet Sample Metricbeat dataset
- 27. Data Rollups
- 28. Data Rollups Supported metrics min max countavgsum
- 29. Data Rollups Supported metrics cardinality percentiles min max countavg
- 30. Data Rollups Flexible bucketing and filtering by time, histograms, and terms prod-1.myco.com prod-2.myco.com prod-3.myco.com prod-4.myco.com prod-5.myco.com Date Histogram Histogram Terms
- 31. Data Rollups @timestamp datacenter url.path Flexible bucketing and filtering by time, histograms, and terms
- 32. Data Rollups @timestamp datacenter url.path Flexible bucketing and filtering by time, histograms, and terms
- 33. Data Rollups @timestamp datacenter url.path Flexible bucketing and filtering by time, histograms, and terms
- 34. Data Rollups @timestamp group by datacenter url.path Flexible bucketing and filtering by time, histograms, and terms
- 35. Data Rollups @timestamp datacenter url.path Flexible bucketing and filtering by time, histograms, and terms
- 36. Data Rollups Flexible bucketing and filtering by time, histograms, and terms @timestamp datacenter filter by url.path
- 37. Data Rollups The more data you have, the more space you save, easily 90%+ Raw data
- 38. Data Rollups
- 39. Search
- 41. Scaleable Cross Cluster Search Elasticsearch Kibana ElasticsearchElasticsearch
- 42. Scaleable Cross Cluster Search Search across two major versions 5.latest 7.x6.latest Elasticsearch Kibana ElasticsearchElasticsearch
- 45. Adaptive Replica Selection Avoids nodes with higher latency 2R2P1R1P
- 46. SQL Client SELECT course, avg(age),count(*) FROM mytable WHERE match(uni,"oxford") GROUP BY course ORDER BY course, avg(age) HAVING avg(age) > 18
- 47. SQL Client CLI JDBC Kibana Canvas SQL over RESTGET /_sql {}
- 48. SQL Client CLI JDBC Kibana Canvas SQL over RESTGET /_sql {} ODBC
- 49. Security
- 50. Secure All the Things Mandatory TLS between nodes
- 51. Secure All the Things Optional TLS across stack
- 52. Secure All the Things changeme changeme changeme No default passwords changeme
- 53. Secure All the Things No plaintext passwords
- 54. Single Sign On with SAML OAuth and Kerberos to follow
- 55. Kibana Spaces
- 56. Kibana Spaces
- 58. Kibana Spaces Application privileges per space
- 59. Administration
- 61. Rolling Major Version Upgrades 5.2 5.2 5.2 5.25.25.6 5.6 5.6 5.6 5.6
- 62. Rolling Major Version Upgrades Upgrade Assistant 5.6 5.6 5.6 5.65.66.x 6.x 6.x 6.x 6.x
- 63. 6.x 6.x 6.x 6.x 6.x Rolling Major Version Upgrades Zero Downtime
- 64. Segment 1 File-Based Recovery (5.x) Primary Segment 2 Segment 3 Segment 1 Replica Segment 2 Segment 3
- 65. Segment 1 File-Based Recovery (5.x) Primary Segment 2 Segment 3 Segment 1 Replica Segment 2 Segment 3
- 66. Segment 4 Segment 1 File-Based Recovery (5.x) Primary Segment 1 Replica Segment 2 Segment 3
- 67. Segment 4 Segment 1 File-Based Recovery (5.x) Primary Segment 1 Replica Segment 4
- 68. Segment 4 Segment 1 File-Based Recovery (5.x) Primary Segment 1 Replica Segment 4
- 69. 5 6 7 Ops-Based Recovery (6.0) Primary Replica 1 2 3 4 5 6 7 1 2 3 4
- 70. 1 2 3 4 5 6 7 Ops-Based Recovery (6.0) Primary Replica 1 2 3 4 5 6 7
- 71. Cross-Cluster Replication New YorkTokyo London ny_sales ny_sales lnd_salestk_sales
- 72. Index Shrinking 1 2 3 4
- 73. Index Shrinking 1 2 3 4
- 78. Frozen Indices • For storing and searching old data • Low heap usage • Frozen indices opened and searched sequentially • Replicated - no data loss Trade disk storage for search latency
- 80. Index Lifecycle Management Hot Phase - Index to my-logs, Search on my-logs
- 81. Index Lifecycle Management Hot Nodes 1 2 3 Cold Nodes Hot Phase - Index to my-logs, Search on my-logs Warm Nodes
- 82. Index Lifecycle Management 1 2 3 Hot Phase - Rollover 1 2 3 Hot Nodes Warm Nodes Cold Nodes
- 83. Index Lifecycle Management 1 2 3 Warm Phase - Allocate 1 2 3 Hot Nodes Warm Nodes Cold Nodes
- 84. 23 Index Lifecycle Management 1 2 3 Warm Phase - Shrink 1 Hot Nodes Warm Nodes Cold Nodes
- 85. 1 Index Lifecycle Management 1 2 3 Warm Phase - Compress Hot Nodes Warm Nodes Cold Nodes
- 86. 1 Index Lifecycle Management 1 2 3 Cold Phase - Allocate Hot Nodes Warm Nodes Cold Nodes
- 87. Index Lifecycle Management 1 2 3 Cold Phase - Freeze Hot Nodes Warm Nodes Cold Nodes 1
- 88. Index Lifecycle Management 1 2 3 Delete Phase Hot Nodes Warm Nodes Cold Nodes 1
- 89. Index Lifecycle Management (coming soon to X-Pack)
- 90. GIS