Implement DevOps Like a Unicorn—Even If You’re Not One
- 1. DW3 Adopting DevOps 6/7/2017 1:30:00 PM DW3 Implement DevOps Like a Unicorn-Even If You're Not One Presented by: Mason Leung Grand Rounds Brought to you by: 350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐268-‐8770 ·∙ 904-‐278-‐0524 - [email protected] - https://www.techwell.com/
- 2. Mason Leung Grand Rounds A backend developer with an interest in infrastructure and operations, Mason Leung is an infrastructure engineer at Grand Rounds, a health tech company that enables patients better access to care providers. In his career roles, Mason has experience defining and establishing a DevOps foundation and culture. Mason is passionate about enhancing developer efficiency and experience through automation, better tooling, and education. On weekends he mentors high school students who are interested in math and STEM. Mason's lifetime goal is to start a tutoring center and provide free lessons to disadvantaged students in San Francisco. Reach Mason at [email protected].
- 3. 1 Implement DevOps Like A Unicorn Even You Don't Work At One
- 4. 2 | © 2017 Grand Rounds, Inc. Background • Mason Leung • Infrastructure Engineer
- 5. 3 | © 2017 Grand Rounds, Inc. "What do you do?" Asked by my 7 years old niece
- 6. 4 | © 2017 Grand Rounds, Inc. Implement DevOps Like A Unicorn Even You Don't Work At One
- 7. 5 | © 2017 Grand Rounds, Inc. What is an unicorn? • Wikipedia: Company valued over $1B • 2013 Aileen Lee, VC
- 8. 6 | © 2017 Grand Rounds, Inc. No Unicorn
- 9. 7 | © 2017 Grand Rounds, Inc. Forrest vs Tree vs
- 10. 8 | © 2017 Grand Rounds, Inc. Scenario Question
- 11. 9 | © 2017 Grand Rounds, Inc. What would you do?
- 12. 10 | © 2017 Grand Rounds, Inc. Plan of Attack The goal as an infrastructure engineer is to identify components of the platform whether it is a process or resources that are potential bottlenecks, figure a way to break them.
- 13. 11 | © 2017 Grand Rounds, Inc. Implicit Goal Bus Factor N to N+1 Hopefully N != 1
- 14. 12 | © 2017 Grand Rounds, Inc. Let's Begin Components to increase scalability and agility (from the infrastructure perspective)
- 15. 13 | © 2017 Grand Rounds, Inc. Deployment
- 16. 14 | © 2017 Grand Rounds, Inc. Deployment Loosely defined term
- 17. 15 | © 2017 Grand Rounds, Inc. Deployment • Resource Provision • Configuration Management • Code Deploy
- 18. 16 | © 2017 Grand Rounds, Inc. Resource Provision • How do I set up the platform? • Is it in the Cloud?
- 19. 17 | © 2017 Grand Rounds, Inc. Lots of Questions • What are the needed resources? • What are the security groups and access control? • What are the VPN access on premise and remote? • What to do with persistent data?
- 20. 18 | © 2017 Grand Rounds, Inc.
- 21. 19 | © 2017 Grand Rounds, Inc. Codify your infrastructure
- 22. 20 | © 2017 Grand Rounds, Inc.
- 23. 21 | © 2017 Grand Rounds, Inc. Is there a DR in the house?
- 24. 22 | © 2017 Grand Rounds, Inc. Terraform: First Try • One huge file • Good for proof of concept • Good learning experience • Problems: organization and collaboration
- 25. 23 | © 2017 Grand Rounds, Inc. Terraform: Second Try • Break down components • Separation of environments • Problem: organization and collaboration
- 26. 24 | © 2017 Grand Rounds, Inc. Terraform: Third Try • Use Terragrunt for state locking • DRY with module
- 27. 25 | © 2017 Grand Rounds, Inc. Considerations • Terraform by applications vs one repo for all applications • Module organization • functional vs flat
- 28. 26 | © 2017 Grand Rounds, Inc. The Benefits • Infrastructure reproducibility • Documentation
- 29. 27 | © 2017 Grand Rounds, Inc. What separates a good programmer and an excellent programmer?
- 30. 28 | © 2017 Grand Rounds, Inc. Excellent programmer documents
- 31. 29 | © 2017 Grand Rounds, Inc. Codify Your Infrastructure • Flexibility and reproducibility • Documentation • Traceability of changes • A step toward DR
- 32. 30 | © 2017 Grand Rounds, Inc. Configuration Management
- 33. 31 | © 2017 Grand Rounds, Inc. In The Past • apt-get install manually • shell scripts
- 34. 32 | © 2017 Grand Rounds, Inc. Which one?
- 35. 33 | © 2017 Grand Rounds, Inc. A Third Option?
- 36. 34 | © 2017 Grand Rounds, Inc. Benefits • Environmental Attributes
- 37. 35 | © 2017 Grand Rounds, Inc. Benefits • Integration with other tools
- 38. 36 | © 2017 Grand Rounds, Inc. Benefits • Shorten debug loop
- 39. 37 | © 2017 Grand Rounds, Inc. Benefits • Community cookbooks or manifests • Lock down version and portability
- 40. 38 | © 2017 Grand Rounds, Inc. Benefit • Documentation
- 41. 39 | © 2017 Grand Rounds, Inc. Continuous Delivery No longer a luxury, but a necessity
- 42. 40 | © 2017 Grand Rounds, Inc. Concepts • Continuous Integration • constantly merging • Continuous Delivery • constantly deploying
- 43. 41 | © 2017 Grand Rounds, Inc. The Old Deploy Pipeline
- 44. 42 | © 2017 Grand Rounds, Inc. The Old Pipeline • Mason Leung • Infrastructure Engineering @ GrandRounds
- 45. 43 | © 2017 Grand Rounds, Inc. Short Term Fix Automate commit messages to slack with gem
- 46. 44 | © 2017 Grand Rounds, Inc. Short Term Fix Score =( Number of commits ) x (Hours since deploy)
- 47. 45 | © 2017 Grand Rounds, Inc.
- 48. 46 | © 2017 Grand Rounds, Inc. Considerations on CD • Buy in from managers and developers • Homogenous deploy tool • In house vs hosted solution or both
- 49. 47 | © 2017 Grand Rounds, Inc. Considerations on CD • Monitoring core metrics • Run Jenkins backup • Modular services • Application and infrastructure cadence
- 50. 48 | © 2017 Grand Rounds, Inc. A Very Important Lesson
- 51. 49 | © 2017 Grand Rounds, Inc. Do NOT build customized tools unless you absolutely have to
- 52. 50 | © 2017 Grand Rounds, Inc. Case 1 • EC2 Creator • Bring up EC2 in console • Use knife-ec2 instead • Run chef configuration
- 53. 51 | © 2017 Grand Rounds, Inc. Problems • Works great for EC2 only • Our platform has other resources • A bigger picture
- 54. 52 | © 2017 Grand Rounds, Inc. Are we in the business of writing ops tools?
- 55. 53 | © 2017 Grand Rounds, Inc. Lessons • Think about your value proposition • Google for tools • Ask questions on mailing list or /r/devops • Documentation
- 56. 54 | © 2017 Grand Rounds, Inc. Case 2 • Home grown CI system • build under 15 minutes • parallelization with containers • split up test with profiler
- 57. 55 | © 2017 Grand Rounds, Inc. Building a CI System From Scratch is Fun
- 58. 56 | © 2017 Grand Rounds, Inc. Maintaining a CI System is Not Fun
- 59. 57 | © 2017 Grand Rounds, Inc. The Trouble Continue • Build start to run more than 15 minutes • Increases parallelism • Cost of CI vs production
- 60. 58 | © 2017 Grand Rounds, Inc. The Real Issues • Poorly written nested tests • New code makes requests to DB • Not enough tests ran
- 61. 59 | © 2017 Grand Rounds, Inc. Lessons Learned • Keep good relationship • Understand the problem beneath • Pay down technical debt • Look at non-functional features
- 62. 60 | © 2017 Grand Rounds, Inc. Building Customized Solutions • Always ask • What is the real problem? • Are there existing solutions? • Enough support within the org?
- 63. 61 | © 2017 Grand Rounds, Inc. Your Great Customized Solution Is A Snowflake
- 64. 62 | © 2017 Grand Rounds, Inc. If You Insist...
- 65. 63 | © 2017 Grand Rounds, Inc. So Far • Resource provisioning • Configuration management • Code deployment • Use existing tools • Resists making snowflakes • Documentation
- 66. 64 | © 2017 Grand Rounds, Inc. Putting All Your Eggs In One Basket
- 67. 65 | © 2017 Grand Rounds, Inc. Multiple Vendors • DNS: DNSMadeEasy • CDN: EdgeCast • Other resources: AWS
- 68. 66 | © 2017 Grand Rounds, Inc. A Big Migration • VPC migration • DNS and CDN consolidation • Finish before the holiday season starts
- 69. 67 | © 2017 Grand Rounds, Inc. Three Phrases • DNS Migration • CDN Migration • VPC Migration
- 70. 68 | © 2017 Grand Rounds, Inc. DNS Migration • Create new host zone in R53 • Update domain registrar • Export zone file in DNSME and reimport on R53 • Monitor queries
- 71. 69 | © 2017 Grand Rounds, Inc. Redirect Problem • Nifty redirect feature • blog.example.com to example.com/blog • vpn.example.com to http://1.2.3.4
- 72. 70 | © 2017 Grand Rounds, Inc. Redirect Problem Solution • Do it through S3 • Run your own redirect service • Third party
- 73. 71 | © 2017 Grand Rounds, Inc. Third Party Redirect Not a lot of choices for URL redirect
- 74. 72 | © 2017 Grand Rounds, Inc. Lesson Learned • Ask questions • Look at nifty features • Be transparent • Migration goes according to plan
- 75. 73 | © 2017 Grand Rounds, Inc. CDN Migration • Weighted policy in Route 53 • Two type of contents • S3 • Assets on EC2 behind ELB
- 76. 74 | © 2017 Grand Rounds, Inc. Migration Steps • Enable CloudFront for either a S3 bucket or an ELB • Forward the HOST header to enable SSL in CloudFront • Split CDN reference into two in Route 53 • Dial up traffic slowly • Repeat
- 77. 75 | © 2017 Grand Rounds, Inc. A Redirect Loop • A catch all /platform/ regex left in route • DDos-ing ourselves platform.example.com sfa.example.com sfp.example.com
- 78. 76 | © 2017 Grand Rounds, Inc. Overload our ELB • ELB not warm enough • Call Amazon ahead of time to warm up
- 79. 77 | © 2017 Grand Rounds, Inc. Lessons Learned • Understand your usage pattern, watch for capacity issue • Always have another pair of eyes • Monitor core metrics
- 80. 78 | © 2017 Grand Rounds, Inc. VPC Migration • Beginning of October • Smoother than expected
- 81. 79 | © 2017 Grand Rounds, Inc. Clean Up and Review • Review our architecture • Design with 10x growth • Using internal ELBs • Rewrite pingdom checks • Copy RDS backups with Lambda
- 82. 80 | © 2017 Grand Rounds, Inc. Implement DevOps Like A Unicorn Even You Don't Work At One
- 83. 81 | © 2017 Grand Rounds, Inc. Principles • Transparency • Repeatability • Codify your infrastructure • Use configuration management tools • Documentation • Agility • Have a CD pipeline • Monitoring • Identify core metrics • Resist snowflakes • Don't reinvent the wheel
- 84. 82 | © 2017 Grand Rounds, Inc. FIN