Elastic Stack roadmap deep dive
3 likes1,173 views
- Elastic provides a search and analytics platform called the Elastic Stack that includes the Elastic Stack, Beats data shippers, and Kibana analytics and visualization tools. - The presentation discussed updates to Elastic's products including performance improvements to search, new features for distributed search across data centers, and enhanced security options for authentication and authorization. - Elastic aims to provide customizable and extensible solutions for users to ingest, store, search, analyze and visualize large volumes of data from various sources.
!["hits": {
"total": 123456789,
"hits": [ ... ]
}
"hits": {
"total": {
"value": 10000,
"relation": "gte"
},
"hits": [ ... ]
}
Weak-AND](https://image.slidesharecdn.com/elasticontourstacksep2019final1-191004161700/85/Elastic-Stack-roadmap-deep-dive-68-320.jpg)
![Templated Alerts
when [CPU] > [90%]
then alert
[alerts@me.com]](https://image.slidesharecdn.com/elasticontourstacksep2019final1-191004161700/85/Elastic-Stack-roadmap-deep-dive-113-320.jpg)
Elastic Stack roadmap deep dive
- 1. Shay Banon | Founder & CEO Aaron Katz | CRO Janesh Moorjani | CFO Elastic Overview Elastic Stack Updates and Roadmap Alex Francoeur
- 5. Ingest
- 7. Ingest
- 8. Ingest
- 9. Ingest
- 10. Ingest
- 12. Ingest Node: Enrichment Processor source.ip => is_known_botnet? geo.location => city/region/country
- 13. Adding Data
- 15. • What technology? (eg. Nginx) • What to monitor? (eg. logs, metrics, packets) • Where is it? (eg. paths to logfiles) Integrations Manager
- 18. • Beats config • Ingest node config • Index template • First index • Index alias • Index lifecycle management policy • Snapshot lifecycle management policy • Index patterns • Kibana dashboards • Canvas workpads • Machine learning jobs • Alerts Automatically Setup
- 19. • Beats config • Ingest node config • Index template • First index • Index alias • Index lifecycle management policy • Snapshot lifecycle management policy • Index patterns • Kibana dashboards • Canvas workpads • Machine learning jobs • Alerts Automatically Setup
- 21. • Single config language • Installs required Beats • Upgrades Beats • Upgrades itself Beats Agent
- 22. Fleet • Centralized Config Deployment • Centralized Beats Monitoring • Centralized Upgrade Management
- 23. Data Management
- 24. Frozen Indices
- 25. Heap File system cache Disk
- 26. Heap File system cache Disk
- 28. Hot Nodes 1 2 3 Cold NodesWarm Nodes
- 29. 1 2 3 1 2 3 Hot Nodes Cold NodesWarm Nodes
- 30. 1 2 3 1 2 3 Hot Nodes Cold NodesWarm Nodes
- 31. 231 2 3 Hot Nodes Cold NodesWarm Nodes 1
- 32. 1 Hot Nodes Cold NodesWarm Nodes 1 2 3
- 33. 1 Hot Nodes Cold NodesWarm Nodes 1 2 3
- 34. Hot Nodes Cold NodesWarm Nodes 1 2 3
- 35. Hot Nodes Cold NodesWarm Nodes 1 2 3
- 36. (coming soon to X-Pack)
- 38. • Periodic scheduled backups • Retention polices for automatic deletion Snapshot Management
- 41. Data Transforms
- 42. Clickstream Data
- 43. Page views per minute? Clickstream Data
- 44. 99th percentile latency? Clickstream Data
- 45. Most frequent URLs? Clickstream Data
- 46. How long was session 1? Clickstream Data
- 47. How long was session 1? Clickstream Data
- 48. Average session length? Clickstream Data
- 49. Average session length? Session Data
- 50. Average number of pages per session? Session Data
- 51. Most frequent exit page per session? Session Data
- 52. Session Data
- 53. How frequently do users visit the site? Session Data
- 54. How frequently do users visit the site? User Data
- 55. • Pivot • Pattern Matching Data Transformation
- 58. • Outlier detection • Supervised model training for regression & classification • Ingest Prediction Processor Advanced ML Analytics
- 59. Search
- 61. Query Before After Improvement Fuzzy 46 qps Phrase 4 qps Bool AND 9.3 qps Bool OR 3.3 qps Term 33 qps
- 62. Query Before After Improvement Fuzzy 46 qps 59 qps 28% Phrase 4 qps 7 qps 87% Bool AND 9.3 qps 23.5 qps 247% Bool OR 3.3 qps 9.8 qps 292% Term 33 qps 1,160 qps 3,700%
- 63. Magic WAND
- 64. "query" : "elasticsearch and lucene" max_score(and) == 1 max_score(lucene) == 5 max_score(elasticsearch) == 3 Weak-AND
- 65. Min top-10 score and (1) elasticsearch (3) lucene (5) <=1 ✓ ✓ ✓ > 1 and <= 4 ✗ ✓ ✓ > 4 and <= 9 ✗ ✗ ✓ > 9 ✗ ✗ ✗ Weak-AND
- 66. Weak-AND
- 67. Weak-AND "aggs": { ... } "track_total_hits": true
- 68. "hits": { "total": 123456789, "hits": [ ... ] } "hits": { "total": { "value": 10000, "relation": "gte" }, "hits": [ ... ] } Weak-AND
- 69. Search as you type
- 70. index_prefixes: qu, qui, quic, quick br, bro, brow, brown fo, fox, foxe, foxes
- 71. index_phrases: the_quick quick_brown brown_fox fox_jumped jumped_over over_the the_lazy lazy_dog
- 73. Advanced Scoring
- 74. rank_feature: Advanced Scoring • Star Ratings • PageRank • Popularity
- 75. score = BM25(Text) + PageRank rank_feature: Advanced Scoring • Star Ratings • PageRank • Popularity
- 76. score = BM25(Text) + Saturation(PageRank) rank_feature: Advanced Scoring • Star Ratings • PageRank • Popularity
- 77. distance_feature: rank_feature: Advanced Scoring • Date • Geopoint • Numeric • Star Ratings • PageRank • Popularity
- 78. script_score: • Custom scoring, including vectors distance_feature: • Date • Geopoint • Numeric rank_feature: Advanced Scoring • Star Ratings • PageRank • Popularity
- 79. Work with WAND script_score: • Custom scoring, including vectors distance_feature: rank_feature: Advanced Scoring • Star Ratings • PageRank • Popularity • Date • Geopoint • Numeric
- 80. Result Pinning
- 82. Geoshapes
- 83. • v2.3: 1 dim, for numbers and dates • v5.0: 2 dim, for geopoints • v5.2: 2 dim, for number & date ranges • v6.7: 7 dim, for geoshapes BKD Trees
- 84. BKD Geoshapes
- 85. • Accurate to 1cm, vs 50m • Index is 60% smaller • Indexing 60% faster • Queries 50% faster • Plus BKD GeoPoints 80% faster indexing BKD Geoshapes
- 87. Zen
- 94. New York London Tokyo
- 95. v5.6 v6.7 v7.x Three Major Versions
- 97. New York London Tokyo ldn_sales ldn_sales
- 98. New York London Tokyo tk_salesny_sales
- 99. New York London Tokyo tk_salesny_sales ldn_sales ldn_sales
- 101. Kibana
- 102. Security
- 106. Lens
- 108. New Platform
- 111. SIEM Stack Monitoring Machine Learning Observability
- 112. 112
- 113. Templated Alerts when [CPU] > [90%] then alert [[email protected]]
- 114. Chart-based Alerts
- 116. Guides
- 118. News Feed
- 120. Thank you