Serverless Logging with AWS Lambda and the Elastic Stack
Download as PPTX, PDF0 likes793 views
The document discusses serverless logging with ELK (Elasticsearch, Logstash, Kibana) for monitoring Lambda functions. It provides an overview of serverless architecture, why it is useful for irregular traffic that auto-scaling cannot handle, and challenges like vendor lock-in. Typical use cases are real-time processing of files, streams, ETL, IoT backends, and APIs. The demo will show monitoring Lambda functions that generate Game of Thrones plot points using ELK for logging and monitoring.
Serverless Logging with AWS Lambda and the Elastic Stack
- 1. SERVERLESS LOGGING WITH ELK Edoardo Paolo Scalafiotti - Software Engineer @ Sainsbury’s
- 3. About me Go, NodeJS SPAs, WebComponents Material Design Serverless Architecture Microservices Machine Learning FabLab, Meetups
- 6. TL;DR FaaS is about running back end code without managing your own server systems or your own server applications
- 7. applications that run in stateless compute containers that are event-triggered, ephemeral and fully managed by a 3rd party
- 8. ● 2 overlapping areas: BaaS and PaaS ● run code without provisioning or managing servers ● horizontal scaling is completely automatic, elastic, and managed by the provider ● responds to triggers
- 10. WHY SHOULD WE CARE?
- 12. Typical servers in business and enterprise data centers deliver between 5 and 15 percent of their maximum computing output on average over the course of the year
- 13. ● Reduced operational cost ● Good with irregular, high spike traffic that cannot be handled by auto-scaling ● Promotes best coding practises
- 14. WHAT ARE THE CHALLENGES?
- 15. TL;DR not for the faint-hearted... at the moment
- 16. ● Ugly hacks for keeping the function alive ● Vendor lock-in ● Unit testing is ok, integration... another story ● Repetition of logic ● Concurrent limit ● Logging, Monitoring and Debugging
- 17. USE CASES
- 18. USE WISELY FaaS, containers and VMs are not enemies
- 19. FAAS is seen as a better choice for event-driven apps with few event types per application component, while CONTAINERS are seen as a better choice for synchronous-request driven components with many entry points
- 20. VM Containers Serverless Configure machines, storage, networking and OS Run servers, configure applications and control scaling Run my code when it’s needed
- 21. Use cases: ● Real-time file processing (ingestion) ● Real-time stream processing (mkt. data) ● Extract, Transform, Load ● IoT Backends ● Mobile Backends ● RESTful APIs
- 22. ● Open Source alternative: IBM’s OpenWhisk ● API Orchestration: serverless.com and claudia.js ● Multiple PL support: Apex ● Logging and monitoring: ELK
- 23. LET’S GET TO THE CODE...
- 24. THE MISSION ...is to monitor a set of Lambda-based functions that generates Game Of Thrones plot points
- 26. DEMO
Editor's Notes
- #2: This talk is divided in 2: Introduction on What is Serverless, Serverless logging with ELK, demo in NodeJS and GOlang
- #3: To summerize, this talk is about why I think that ELK is a good solution - not the only one… - for monitoring logs and metrics for Serverless
- #6: When we talk about serverless we are mainly talking about AWS’ Lambda, which is the more mature solution out there, released in 2014. Other vendors are coming with similar solutions like: Google Cloud Functions (Alpha) Azure Functions PL support: Lambda: Python, Java (slow) and NodeJS Google Cloud: NodeJS Azure: Javascript, C#, Python, PHP We’ll talk about Lambda
- #17: No offline testing Concurrent limit is 100 functions, then queuing / throttling happens
- #19: There is the perception that VMs, Containers and Serverless are equivalent alternatives Although they represent higher layers of abstraction VM abstracts the hardware Container the OS Serverless the language runtime They are suited for different use cases. This is about having more choices, not the next hot thing