AWS re:Invent re:Cap 2019: My ElasticSearch Journey on AWS
1 like586 views
The document outlines a presentation on the author's journey with Elasticsearch on AWS, discussing the challenges of maintaining high availability, scaling, and cost management. It highlights experiences with both self-hosted and AWS Elasticsearch services, Kubernetes deployments, and the use of Open Distro. The author shares insights into the advantages and drawbacks of each approach, alongside financial comparisons and compliance considerations.
AWS re:Invent re:Cap 2019: My ElasticSearch Journey on AWS
- 1. TAIPEI
- 2. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. My ElasticSearch Journey on AWS smalltown Lead Site Reliability Engineering AMIS 2
- 3. Hello! I am smalltown AMIS Lead Site Reliability Engineering Taipei HashiCorp User Group Organizer AWS User Group Taiwan Staff 3
- 4. Agenda ● Background ● ElasticSearch on Private Cloud ● AWS ElasticSearch Service ● ElasticSearch Operator on Kubernetes ● Open Distro for ElasticSearch on Kubernetes 4
- 5. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Background 5
- 6. Life is Hard, Not Mention to Maintain ELK High Availability Scalability Index Backup Retention Policy Health Status Monitoring 6
- 7. My Elastic Decision Tree Service Self-Hosted Open Distro 7
- 8. My Kubernetes Decision Tree Self-Hosted AWS EKS Spot On Demand 8
- 9. What is My ElasticSearch Journey?! ● ElasticSearch on Private Cloud ● AWS ElasticSearch Service ● ElasticSearch Operator on Kubernetes ● Open Distro for ElasticSearch on Kubernetes 9
- 10. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. ElasticSearch on Private Cloud 10
- 11. What Happened?! ● First Time Maintain Official ElasticSearch Cluster ● Manually Scale Instance Number & Type, also Disk Volume, IOPS ● Don’t Know Cost How Much (Chargeback) 11
- 12. Retrospective Advantage The Cost of Large-Scale Data Center is Lower than Public Cloud Soft Reminder Cannot Scale Resource Automatically, Need to colleagues’ assistant 12
- 13. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS ElasticSearch Service 13
- 14. What Happened?! ● There is no Time for a Crazy Growing Startup Company to Survey, Setup ● ElasticSearch Service Provide a Fast, Stable Way for Log Central Management ● Only One Click, Everything Works 14
- 15. Retrospective Advantage Don’t need to Waste Time to Setup ElasticSearch Soft Reminder Comparing to Self-Hosted ElasticSearch, The Cost is Higher 15
- 16. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. ElasticSearch Operator on Kubernetes 16
- 17. What Happened?! ● The Log Size Grows Day by Day, Hence, The Cost also Grows ● Some Teams Only Use Kubernetes as Application Workload ● Find the Easy Way to Deploy ElasticSearch into Kubernetes (Operator) 17
- 18. What is ElasticSearch (K8S) Operator? ● It is a Software with ElasticSearch Operation Knowledge ● Enable Users to Create, Configure, and Manage ElasticSearch Cluster Automatically Ref18
- 19. Retrospective Advantage Leverage Vishwakarma to Self-Hosted Kubernetes Cluster W/ All Spot Instances Leverage ElasticSearch Operator to Deploy ElasticSearch Cluster within Several Hours Soft Reminder The Log Size Still Grows, Operator Cannot Adjust Detail ElasticSearch Configuration ElasticSearch OSS Version Doesn’t Fulfill Company’s Compliance 19
- 20. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Open Distro for ElasticSearch on Kubernetes 20
- 21. What Happened?! ● Still Using Vishwakarma to Self-Hosted Kubernetes Cluster W/ All Spot Instances ● Leverage Elastic Official Helm Chart W/ Little Customize to Deploy ElasticSearch & Kibana ● Replace ElasticSearch & Kibana OSS W/ Open Distro for ElasticSearch 21
- 22. What is Open Distro for ElasticSearch? ● AWS Official Depends on ElasticSearch OSS version to Append More Strong Features ● Currently There are Security, Alerting, SQL, Performance Analyzer, Index Management...etc ● Preparing to add Anomaly Detection, k-NN Search, Security integration with alerting in the future 22
- 23. Retrospective Advantage Elastic Official Helm Chart Let me do whatever I want for the ElasticSearch Cluster The Kubernetes Cluster Still Keeps the Lowest Cost The ElasticSearch Cluster Meet My Company Compliance Now Soft Reminder It takes Time to Setup Open Distro for ElasticSearch (Document is not so Friendly) 23
- 24. Current ELK Infrastructure Beats Kibana Logstash ElasticSearch Vishwakarma Redis LDAP/SAML Open Distro AWS S3 Prometheus 24
- 25. 💰 Money! 💰 Money! 💰 Money! By Month (US$) ElasticSearch (r5.2xlarge * 5) Logstash (t3.large * 2) ETCD/Kibana (t3.medium * 1) K8S Master (t3.medium * 1) Total Service 2,675 120 30 (Kibana) N/A 2,825 Self-Hosted 540 (Spot) N/A 30 (ETCD) 9 (Spot) 579 ● This is just a Simple Example ● Not Including EBS Volume, Redis...etc (Both Solution Necessary) 25
- 26. THANKS! ANY QUESTIONS? You can find me at my office: ● Frontend Engineer ● Software Engineer ● Software Engineer in Test ● Project Manager Job Page 26