How to build a social network on serverless
1 like984 views
The document discusses building a social network using serverless architecture, emphasizing the benefits of microservices, event-driven approaches, and cost efficiency with tools like AWS Lambda and Kinesis. Key takeaways include reducing deployment times, achieving significant cost savings, and improving user experience by delivering features faster. It highlights the importance of organized function management and incrementally migrating legacy systems to modern serverless solutions.
How to build a social network on serverless
- 1. How to build a social network on #serverless Yan Cui @theburningmonk
- 7. apr, 2016
- 8. nov, 2016
- 9. WHY?
- 11. hey guys, vote on this post and I’ll announce a winner at 10PM tonight
- 12. 10PM traffic
- 14. low utilisation to leave room for spikes EC2 scaling is slow, so scale earlier
- 15. lots of $$$ for unused resources
- 16. up to 30 mins for deployment deployment required downtime
- 17. features took months to develop
- 18. - Dan North “lead time to someone saying thank you is the only reputation metric that matters.”
- 20. WHY? to deliver better UX
- 21. WHY? to deliver better UX to deliver value faster
- 22. WHY? to deliver better UX to deliver value faster to be more cost efficient
- 23. WHY? to deliver better UX to deliver value faster to be more cost efficient HOW?
- 24. what would good look like for us?
- 25. small fast zero downtime no lock-step deployments should be…
- 26. features should be… deployable independently loosely-coupled
- 27. we want to… minimise cost for unused resources
- 28. we want to… minimise cost for unused resources minimise ops effort
- 29. we want to… minimise cost for unused resources minimise ops effort reduce tech mess
- 30. we want to… minimise cost for unused resources minimise ops effort reduce tech mess deliver visible improvements faster
- 31. WHY? to deliver better UX to deliver value faster to be more cost efficient HOW? microservices
- 32. WHY? to deliver better UX to deliver value faster to be more cost efficient HOW? microservices event-driven
- 33. WHY? to deliver better UX to deliver value faster to be more cost efficient HOW? microservices event-driven serverless
- 34. WHY? to deliver better UX to deliver value faster to be more cost efficient HOW? microservices event-driven serverless WHAT? this talk!
- 35. WHY? to deliver better UX to deliver value faster to be more cost efficient HOW? microservices event-driven serverless WHAT? this talk!
- 36. 170 Lambda functions in prod
- 37. 95% cost saving vs. EC2
- 38. 15x no. of prod releases per month
- 39. 15x no. of prod releases per month (features were sometimes shipped on the same day)
- 40. time is a good fit
- 41. 1st function in prod! time is a good fit
- 42. ? time is a good fit 1st function in prod!
- 43. CI/CD?
- 44. CI/CD? testing?
- 46. time is a good fit 1st function in prod! CI/CD, testing, logging, monitoring, alerting
- 47. 170 functions ? time is a good fit 1st function in prod! CI/CD, testing, logging, monitoring, alerting
- 48. tracing?
- 51. 170 functions time is a good fit 1st function in prod! CI/CD, testing, logging, monitoring, alerting tracing, config management, security
- 52. API Gateway and Kinesis Authentication & authorisation (IAM, Cognito) Testing Running & Debugging functions locally Log aggregation Monitoring & Alerting X-Ray Correlation IDs CI/CD Performance and Cost optimisation Error Handling Configuration management VPC Security Leading practices (API Gateway, Kinesis, Lambda) Canary deployments http://bit.ly/production-ready-serverless get 40% off with: ytcui
- 54. Legacy Monolith Amazon Kinesis Step 1. ALL state changes!
- 55. events are an enabler for COMPOSABILITY
- 57. Kinesis
- 58. Kinesis API Gateway AWS Lambda API GatewayAWS Lambda service-A service-B
- 59. Kinesis API Gateway AWS Lambda API GatewayAWS Lambda service-A service-B
- 60. Kinesis API Gateway AWS Lambda API GatewayAWS Lambda service-A service-B AWS Lambda AWS Lambda AWS Lambda
- 61. Kinesis API Gateway AWS Lambda API GatewayAWS Lambda service-A service-B AWS Lambda AWS Lambda AWS Lambda DynamoDBIOT
- 62. Kinesis API Gateway AWS Lambda API GatewayAWS Lambda service-A service-B AWS Lambda AWS Lambda AWS Lambda DynamoDBIOT
- 63. Kinesis API Gateway AWS Lambda API GatewayAWS Lambda service-A service-B AWS Lambda AWS Lambda AWS Lambda DynamoDBIOT AWS Lambda AWS Lambda
- 64. build loosely-coupled system through events
- 65. service A service B service C service D bounded context bounded context
- 66. service A service B service C service D bounded context bounded context
- 67. service A service B service C service D
- 69. there are no silver bullets
- 70. service A service B service C service D
- 71. service A service B service C service D
- 72. service A service B service C service D update!
- 73. service A service B service C service Dbackward-compatible? update!
- 75. bounded context DON’T use events to orchestrate workflows within the same bounded context
- 76. bounded context adds unnecessary complexity to logging, tracing, and end-to-end reporting
- 77. bounded context the workflow doesn’t exist as a standalone concept, but as the sum of a series of loosely connected parts
- 78. Step Functions use Step Functions instead
- 79. Step Functions don’t forget to emit events from the workflow
- 80. Step Functions so others can react to state changes that happened as part of the workflow
- 81. “how do I organize my functions into code repositories?”
- 82. one repo per function?
- 85. monorepo?
- 86. github repo
- 88. one repo per service?
- 96. github repo github repo github repo github repo user-api timeline-api relationship-api search-api
- 97. CI/CD pipeline per service
- 98. functions are deployed together, as a stack
- 99. Strangler Pattern incrementally migrate the legacy system by gradually replacing pieces of functionalities to the new system
- 101. rebuilt search
- 102. Legacy Monolith Amazon Kinesis Amazon Lambda Amazon CloudSearch
- 103. Legacy Monolith Amazon Kinesis Amazon Lambda Amazon CloudSearchAmazon API Gateway Amazon Lambda
- 104. proxy requests from monolith to new service
- 106. expensive ($3000/month) don’t understand our domain JS based query language
- 107. Legacy Monolith Amazon Kinesis Amazon Lambda Google BigQuery
- 108. Legacy Monolith Amazon Kinesis Amazon Lambda Google BigQuery 1 developer, 2 days design production (his 1st serverless project)
- 109. Legacy Monolith Amazon Kinesis Amazon Lambda Google BigQuery “nothing ever got done this fast at Skype!” - Chris Twamley
- 110. - Dan North “lead time to someone saying thank you is the only reputation metric that matters.”
- 113. Kinesis sink
- 114. Kinesis Kinesis Firehose batch Kinesis events
- 115. Kinesis Kinesis Firehose S3 data lake
- 116. Kinesis Kinesis Firehose S3 Glue analyze data schema, catalog data into tables
- 117. Kinesis Kinesis Firehose S3 Athena Glue query engine
- 118. Kinesis Kinesis Firehose S3 AthenaQuickSight Glue visualization, dashboards
- 119. Kinesis Kinesis Firehose S3 AthenaQuickSight Glue no code is required!
- 120. Kinesis Kinesis Firehose S3 AthenaQuickSight Glue no code is required! pay-per-use!
- 121. user action business intelligence
- 122. user action business intelligence
- 125. Problem didn’t work…
- 128. try figure out what’s going on here…
- 131. Rebuilt with Lambda
- 138. built-in retry and DLQ avoid repeating expensive work of fetching mils of relationships
- 142. github repo timeline-api service: timeline-api provider: name: aws runtime: nodejs6.10 stage: dev region: us-east-1 functions: distribute-yubl: … undistribute-yubl: …
- 144. Problem didn’t work…
- 145. “it returns the first 30 users in the database, by creation time…”
- 147. Rebuilt with Lambda
- 148. BigQuery
- 149. BigQuery
- 150. grapheneDB BigQuery
- 151. grapheneDB BigQuery
- 152. grapheneDB BigQuery
- 153. grapheneDB BigQuery mostly built in one sleepless night…
- 155. more Scalable (and scales faster!)
- 156. Cheaper (don’t pay for idle servers)
- 157. Resilience (built-in redundancy and multi-AZ)
- 158. Secure
- 160. idea production greater Velocity from idea to product
- 161. WHY? to deliver better UX to deliver value faster to be more cost efficient