Serverless Logging Architecture
1 like458 views
This document describes building a serverless log analytics platform. It discusses the challenges with conventional logging architectures that require managing servers and have scalability issues. The document then introduces a serverless approach using AWS services like Kinesis, S3, Elasticsearch, and Kibana that allows logging infrastructure to scale infinitely with no server management. Code examples show how to set up logging pipelines to stream logs in real time to storage and analytics using this serverless architecture.
Serverless Logging Architecture
- 1. BUILDING AND SCALING A LOG ANALYTICS PLATFORM - A SERVERLESS APPROACH Narendran
- 2. A BIT ABOUT ME… Naren Product Engineer Scaling A.I to millions @ MadStreetDen Python, Golang, FOSS, Cycling, Travel
- 3. TWITTER & GITHUB @DudeWhoCode WEBSITE & BLOG www.dudewho.codes
- 5. ARCHITECTING A LOGGING FRAMEWORK
- 6. BEHAVIOURS OF DIFFERENT MICROSERVICES • Mostly Idle • Horizontally scales up • Rapid scale down • Surges
- 7. EXPECTATIONS FROM ANY LOG FRAMEWORK • High Performance • Secure • No data Loss/overwriting • Dynamically Scalable • Highly Available • Independent
- 8. KNOWYOUR LOGS • Real time Analysis • Historical Analysis
- 10. PROBLEMS FACED
- 11. THETHING WITH ELK •Threadpool queue size in Elastic Search • Adding Logstash filters • Burning more money, even for idle time
- 12. WHO WILL MONITORTHE SYSTEMTHAT MONITORS OTHER SYSTEMS?
- 13. WHO WILL MONITORTHE SYSTEMTHAT MONITORS OTHER SYSTEMS?
- 14. WHATSTHE COMMONTHINGTHAT ALL THE COMPONENTS HAVE?
- 15. WHATSTHE COMMONTHINGTHAT ALL THE COMPONENTS HAVE? Servers, Servers, Servers
- 17. GOING SERVERLESS (buzzword alert!) Serverless computing is a cloud computing execution model in which the cloud provider dynamically manages the allocation of machine resources. Pricing is based on the actual amount of resources consumed by an application, rather than on pre-purchased units of capacity. - Wikipedia
- 18. GOING SERVERLESS No server is easier to manage than “no server” - Werner Wogels, CTO,Amazon
- 19. GOING SERVERLESS
- 22. AWS SERVERLESS
- 23. AWS SERVERLESS
- 24. AWS SERVERLESS
- 26. •The client application from which we will get our logs
- 27. • Delivers real-time streaming data to other services such as Amazon S3, Elastic Search. • Configurable producers and consumers
- 28. • Run code without provisioning servers • Event driven, Highly scalable
- 29. • Store data at massive scale • Storage tier for many serverless applications
- 30. • Serverless interactive query service • Point to your data source, define the schema, start querying using standard SQL
- 31. • Cloud-powered visualization tool • Perform ad-hoc analysis and get quick business insights
- 32. • Process streaming data in real time with standard SQL in real time • Scales automatically to match throughput
- 33. • Real time log monitoring • Collect and track metrics, log files, trigger alarms
- 34. MANAGINGTHE SERVERLESS MODEL 1. Web Console 2. Code (AWS SDK) 3. CLI
- 35. DEMO
- 36. YOU CAN WATCHTHE FOLLOWING DEMO AS SCREENCAST AT http://bit.ly/serverless-logging
- 45. SUMMARY • Usual logging architecture: took us weeks and certain level of expertise to implement • Serverless architecture: Up and running in minutes • No maintenance, nearly zero developer interventions after deployment • This Faas architecture can scale infinitely for millions of logs from thousands of microservices