DevOpsDaysRiga 2018: Serhat Can - The Rocky Path to Migrating Production Applications to Serverless Architecture
0 likes•365 views
The document discusses the journey of Opsgenie in migrating to a serverless architecture using AWS Lambda, highlighting key milestones from initial integrations to challenges faced with cold starts and scalability. It emphasizes the importance of monitoring and observability, particularly in understanding latency and managing operational costs effectively. Additionally, it provides insights on lessons learned from incidents and the necessity of designing idempotent functions in serverless applications.
DevOpsDaysRiga 2018: Serhat Can - The Rocky Path to Migrating Production Applications to Serverless Architecture
- 1. @srhtcn The Rocky Path To Migrating Production Applications To Serverless Architecture Serhat Can @srhtcn Technical Evangelist @OpsGenie
- 2. @srhtcn Disclaimer: We still love AWS Lambda Serverless Turkey meetup Advocate for Serverless tech Use AWS Lambda in production A brand-new spinoff: Thundra
- 3. @srhtcn Modern incident management platform for operating always-on services ● Plan and prepare for incidents ● Ensure issues are never missed, and the right people are notified ● Gain insights to improve your operational efficiency
- 4. @srhtcn Engineering grew from 3 to 60 people
- 7. @srhtcn The promise of AWS Lambda
- 8. @srhtcn OpsGenie’s Serverless journey 2015 Writing small scale custom integrations At this point, we started leveraging AWS Lambda to help our customer run custom code
- 9. @srhtcn OpsGenie’s Serverless journey 2015 Writing small scale custom integrations At this point, we started leveraging AWS Lambda to help our customer run custom code First production usage Started using AWS Lambda for leveraging async / not business critical jobs such as DynamoDB autoscale 2016
- 10. @srhtcn OpsGenie’s Serverless journey 2015 Writing small scale custom integrations At this point, we started leveraging AWS Lambda to help our customer run custom code First production usage Started using AWS Lambda for leveraging async / not business critical jobs such as DynamoDB autoscale 2016 Service and Incident Management A new customer facing feature running on AWS Lambda integrated with the rest of the code base. 2017
- 11. @srhtcn OpsGenie’s Serverless journey 2015 Writing small scale custom integrations At this point, we started leveraging AWS Lambda to help our customer run custom code First production usage Started using AWS Lambda for leveraging async / not business critical jobs such as DynamoDB autoscale 2016 Service and Incident Management A new customer facing feature running on AWS Lambda integrated with the rest of the code base. 2017 A Spinoff: Thundra Observability for AWS Lambda 2018
- 13. @srhtcn OpsGenie stack Java 8 AWS services including “Serverless” DynamoDB, SQS
- 14. @srhtcn
- 15. @srhtcn Cold start Photo by Chris Marquardt on Unsplash
- 17. @srhtcn Cold start, when Memory size Code size VPC Classpath scan The language choice
- 19. @srhtcn Cold start, the effect Caring about an operational concern which has nothing to do with you Frustrated users because of slow response Paying more money Timeouts in the calling function
- 20. @srhtcn Cold start, the solutions Wait for AWS to improve it Increase memory (and pay more) Lightweight application framework instead of Spring Do some smart warm-up https://medium.com/thundra/dealing-with-cold-starts-in-aws-lambda-a5e3aa8f532
- 21. @srhtcn Scaling Photo by Vladimir Riabinin on Unsplash
- 23. @srhtcn Functions scale nicely until they don’t
- 24. @srhtcn Account level concurrent execution limit Lambda concurrent execution count for non stream based events: events (or requests) per second * function duration
- 25. @srhtcn Account level concurrent execution limit Lambda concurrent execution count for non stream based events: events (or requests) per second * function duration Hard to deal with peaks in request numbers Takes time to increase the limit Functions affect each other’s scalability
- 26. @srhtcn Latency in a third party can bring your whole system down https://read.acloud.guru/does-aws-lambda-keep-its-serverless-marketing-promise-of-continuous-scaling-e990114bb379
- 27. @srhtcn Function level concurrent execution limit Limit the scalability of non-critical functions Reserved capacity is subtracted from the global limit
- 28. @srhtcn Don’t put your functions in a VPC unless you have to You need sufficient IP addresses in your subnet and ENI to scale https://docs.aws.amazon.com/lambda/latest/dg/vpc.html Determine the ENI capacity you need: Concurrent executions * (Memory in GB / 3 GB)
- 29. @srhtcn Use your 6th sense to debug a scaling issue https://docs.aws.amazon.com/lambda/latest/dg/vpc.html
- 30. @srhtcn Photo by Anna on Unsplash Observability
- 31. @srhtcn Fixing “it is slow” is harder in AWS Lambda
- 32. @srhtcn Fixing “it is slow” is harder in AWS Lambda Too many moving pieces No way to attach an agent Even how to send the monitoring data is a discussion point
- 33. @srhtcn Determine the latency in different levels Automatic instrumentation GC, Thread counts & durations, CPU usage details Get the stack trace in case of an error and drill down See logs, traces, and metrics in one view thundra.ioWhat we needed was
- 34. @srhtcn Event driven Photo by Ian Froome on Unsplash
- 35. @srhtcn You got an unexpected bill from AWS?
- 37. @srhtcn Lessons learned: An incident of 40.000$ Avoid infinite retries Monitor and alert for pricing (no pricing metric for AWS Lambda) Think of Cloudwatch cost and sample logs & metrics
- 38. @srhtcn Functions will be triggered more than once. Design idempotent functions considering the trigger type
- 40. @srhtcn Tools can and do help, but they can't make us care. Containers Will Not Fix Your Broken Culture (and Other Hard Truths) https://queue.acm.org/detail.cfm?id=3185224 - Bridget Kromhout