ISTA 2019 - Migrating data-intensive microservices from Python to Go
0 likes80 views
The document discusses migrating data-intensive microservices from Python to Go at Uber, focusing on architecture, performance, and scalability. It outlines the choice of Go for its high performance, static typing, and broad applicability, and compares different migration strategies for services like invoice generation and data storage. Key lessons include the importance of iterative migration to ensure feature parity and performance monitoring.
ISTA 2019 - Migrating data-intensive microservices from Python to Go
- 2. Migrating data-intensive microservices from Python to Go Nikolay Stoitsev Engineering Manager @ Uber
- 3. Early years Dispatch API Storage
- 4. Early years Dispatch API Storage PythonNode.js
- 7. Background Legal document Vary by country
- 8. Background Legal document Vary by country Vary by business line
- 9. Background Legal document Vary by country Vary by business line Triggered after every trip or food delivery
- 10. Sample architecture Money System Cassandra Kafka Preprocess Render Kafka Consumer Object Store
- 11. More than 30 upstream systems Large Scale
- 12. More than 30 upstream systems More than 100 TBs of data stored Large Scale
- 13. More than 30 upstream systems More than 100 TBs of data stored Running on 400 containers in multiple DCs Large Scale
- 14. More than 30 upstream systems More than 100 TBs of data stored Running on 400 containers in multiple DCs Running for 5 years Large Scale
- 15. More than 30 upstream systems More than 100 TBs of data stored Running on 400 containers in multiple DCs Running for 5 years 99.999% availability for last 6 months Large Scale
- 16. More than 30 upstream systems More than 100 TBs of data stored Running on 400 containers in multiple DCs Running for 5 years 99.999% availability for last 6 months Implemented in Python Large Scale
- 17. Sample architecture Money System Cassandra Kafka Preprocess Render Kafka Consumer Object Store Web API Hive
- 19. Flask Example
- 20. Flask Usage
- 21. MVCS
- 23. Building blocks https://uwsgi-docs.readthedocs.io/
- 27. “Use the right tool for the job”
- 28. It hurts velocity at some point
- 29. What we need for each language? Training / Best practices / Documentation / Experts
- 30. What we need for each language? Training / Best practices / Documentation / Experts Project template / Bootstrapping
- 31. What we need for each language? Training / Best practices / Documentation / Experts Project template / Bootstrapping Configuration
- 32. What we need for each language? Training / Best practices / Documentation / Experts Project template / Bootstrapping Configuration Debuggers
- 33. What we need for each language? Training / Best practices / Documentation / Experts Project template / Bootstrapping Configuration Debuggers Profilers
- 34. What we need for each language? Training / Best practices / Documentation / Experts Project template / Bootstrapping Configuration Debuggers Profilers Building, Packaging, Deploying
- 35. We picked Go and Java
- 36. Why Go?
- 38. High performance
- 39. Static typing
- 40. Has momentum
- 41. From Python to Go
- 42. EAFP versus LBYL
- 43. EAFP versus LBYL
- 44. Dependency injection https://github.com/uber-go/fx
- 45. Cadence instead of Celery https://github.com/uber/cadence
- 49. Option #1 - Big Bang Rewrite Money System Invoice Generation StoragePython Invoice Generation Go
- 50. Option #1 - Big Bang Rewrite Money System Storage Invoice Generation Go
- 51. No visibility on regressions
- 52. No visibility on performance degradation
- 53. No visibility on feature parity
- 54. Option #2 - Do it iteratively
- 60. Volume?
- 62. Tally - stats collection in Go https://github.com/uber-go/tally
- 63. Tally - stats collection in Go
- 64. Measure processing time p95, p99
- 66. Correctness?
- 74. Zap
- 75. ELK
- 76. Benefits of iterative approach Verify regressions Verify performance problems Verify feature parity
- 77. Lessons learned
- 78. Spend time to learn the new language
- 79. Spend time to read code in the new language
- 80. Do a rollout plan and stick to it
- 81. Python can scale and is reliable