Sloppy Little Serverless Stories
- 1. From Oops… to Ops Sloppy Little Serverless Stories Sheen Brisals The LEGO Group sheenbrisals
- 2. Oops! A word that is used to show the recognition of a mistake Ops Operations
- 4. LEGO.com was switched to Serverless on AWS Cloud July 10, 2019
- 5. 200+ Lambda functions 40+ microservices 40+ API endpoints 25+ DynamoDB tables 30+ S3 buckets 20+ SNS topics 60+ SQS queues + DLQs 200+ SSM parameters PROD - Services Stats
- 7. No one starts perfect with Serverless. That’s fine, but strive to be better at every iteration, and that is important.
- 9. Kinesis Data Firehose API Gateway S3 bucket Event producer Fan-out function Click-stream event ingestion Buffer size Buffer interval
- 10. Oops! Ops Buffer size: 3 MB Buffer interval: 1 min Peak season lambda failure Buffer size: 1 MB Buffer interval: 1 min Perfect with a tuned lambda
- 11. Oops! 2 When SSM Parameter Store packed a punch…
- 12. 200+ Lambda functions 40+ microservices 40+ API endpoints 25+ DynamoDB tables 30+ S3 buckets 20+ SNS topics 60+ SQS queues + DLQs 200+ SSM parameters PROD - Services Stats
- 13. 200+ Lambda functions 40+ microservices 40+ API endpoints 25+ DynamoDB tables 30+ S3 buckets 20+ SNS topics 60+ SQS queues + DLQs 200+ SSM parameters PROD - Services Stats
- 14. 200+ Lambda functions 200+ SSM parameters PROD - Services Stats Reads Writes
- 15. Oops! Ops Default throughput 40 TPS Rate limiting
- 16. Oops! Ops Default throughput 40 TPS Rate limiting Advanced throughput 1000 TPS Self change via console
- 17. Oops! 3 When too much became too little…
- 19. Oops! Ops Failing at wrong time Missing crucial logs Inadequate testing Incorrect dashboards Better dev process Better monitoring
- 20. Oops! 4 When going higher gone through the roof…
- 21. CloudWatch Event Heavy lifting function Art of coding – Copy & Paste Trigger rule StepFunction 2 GB RAM 5 mins run 2 x daily Frontend Status check API Request handler Status store 2 GB RAM 100 ms run 1000s x daily
- 22. Oops! Ops Memory: 2 GB Invocations: 1 per sec Invocations: 2.5mil /mo Cost~ $9.00 / month Duration: 100 ms Memory: 256 MB Invocations: 1 per sec Invocations: 2.5mil /mo Cost~ $1.50 / month Duration: 100 ms
- 23. Oops! 5 When a key moment turned chaotic…
- 24. SaaS API Feeds Feeds store DLQ Data Pipeline Internal Feeds API Feeds Internal App {API Key} The API re-deployment somehow changed the API Key! No one knew. No one noticed!
- 25. Oops! Ops Silent breakage Bad customer exp. Unhappy business Chaotic dev process Better dev process Better prompts Better monitoring
- 27. Serverless requires a new way of thinking, new way of working, and new way of running applications. That means, we need to change our way of thinking, our way of working, and our way of running applications.
- 30. Oopsibility! 2 Dev to Prod debacle • Keys & secrets • Domains • Resource Configurations • Leaked privacy
- 36. From Oops of Sorrows to Operational Success… • Know the service limits • Be Well-Architected • See through the Serverless Lens • Alarm – Alert – Act • Monitor monitor monitor