DevOps Fest 2020. immutable infrastructure as code. True story.
0 likes172 views
This document discusses the journey of transitioning infrastructure management at Namecheap to an immutable infrastructure as code model using tools like Terraform, Docker, and Jenkins. Key points include taking over a project from an outsourcing company, setting up immutable infrastructure with infrastructure as code, configuring CI/CD pipelines as code in Jenkins, and lessons learned around testing, chaos engineering, and encouraging team feedback. The overall goals were to make infrastructure hard to break, easy to repair, and easy to modify.
![Nowadays: CD
@Library('namecheap/common') _
properties([parameters([
string(name: 'image', description: 'Application container name'),
string(name: 'version', description: 'Application container tag'),
choice(choices: ['production', 'sandbox'], description: 'Environment', name: 'env')
])])
node() {
def authToken = "XXXX"
stage('Deploy') {
deployToOKD(params, authToken)
}
}](https://image.slidesharecdn.com/devopsfest2020-200604181048/85/DevOps-Fest-2020-immutable-infrastructure-as-code-True-story-29-320.jpg)
DevOps Fest 2020. immutable infrastructure as code. True story.
- 1. Immutable Infrastructure as Code. True story. Vladlen Fedosov, Director of R&D @Namecheap, Inc
- 2. Vladlen Fedosov Director of R&D @Namecheap TL;DR: • 10 years in the industry • Went path from Junior to Architect • Amateur DevOps evangelist • AWS ninja • Believe in self-organized, cross-functional teams
- 3. “Opening the door for everyone to a free and open Internet”
- 4. The beginning Disclaimer: Here I’m talking mostly about my experience and part of the infrastructure that my team was responsible for. So further statements may not apply to every department in the company. Timeline for the things mentioned below was changes to simplify storytelling.
- 6. Project takeover from outsourcing company
- 7. Brave New World For a small dev team with no infra support / capabilities
- 8. So what do we have now?
- 9. State of the project after takeover ● Half broken Chef cookbooks ● Sketchy CD pipelines ● Fault tolerance in place ● And… Everything went down after failure of the single (out of 3) etcd node. We realized that we have fault tolerance only on paper
- 10. After some refactoring… We’ve got typical setup
- 11. Issues we noticed - Multiple apps were sharing same OS, language versions, dependencies - Horizontal scaling was hard - Sometimes failing chef scripts on random instances mostly due to network errors & configuration differences
- 12. Configuration Synchronisation problem, in short martinfowler.com/bliki/ConfigurationSynchronization.html
- 13. Issues we noticed - Manual sync of the AWS setup between environments - Manually configured CI/CD - Easy to break something and hard to repair or modify anything
- 14. Blue sky vision ● Immutable infrastructure ● Everything as code ○ Infrastructure as code ○ CI/CD as code
- 15. Blue sky vision ● Hard to break ● Easy to repair ● Easy to modify
- 16. Immutable infrastructure + extra takeaways
- 17. What is immutable infrastructure? martinfowler.com/bliki/ImmutableServer.html
- 18. Why go Immutable? ● Forget about change flow, only “create” matters ● Defeat Configuration Drift ● Use much simpler tools ● Build highly available systems easier ● Fix issues faster
- 19. How to achieve this? ● Complexity → Docker (or AMI) images ● OS → Docker runtime only ● App/OS configuration methods: ○ K8s pod definitions (or similar) ○ “cloud-init” ● Terraform to define “datacenter config”
- 20. Main tools we use ● AMI images & Docker images ● Cloud-Init (98%) ● Hashicorp Packer (2%) ● Terraform, Terraform everywhere
- 21. Single fact you could memorise here Immutable Infrastructure allows you to significantly simplify management steps and consequently reduce number of bugs your customers will face with. Work with images rather than servers.
- 22. CI/CD as code
- 25. Everything as code: full list ● Everything as code, it supports: ○ job steps definition as code (via pipelines) ○ jobs creation as code (via job dsl) ○ system configuration as code (via groovy API, XML configs & CasC yml) ● Shared libraries, ability to share common steps between apps P.S: Talk to me if know better alternative ;)
- 26. Other factors influenced the choice ● Has deployment dashboard so we can see the state of all the environments ● Highly extensible ● Elastic EC2 instances as agents
- 27. Nowadays ● It went beyond one project and now almost every team at Namecheap uses it ● 300+ pipelines ● Around 38 projects ● We expect even more in the future
- 28. Nowadays: CI @Library('namecheap/common') _ node('CommonCLarge') { ciJavascript.servicePipeline { productName = "ProductA" serviceName = "Apps.Api" } }
- 29. Nowadays: CD @Library('namecheap/common') _ properties([parameters([ string(name: 'image', description: 'Application container name'), string(name: 'version', description: 'Application container tag'), choice(choices: ['production', 'sandbox'], description: 'Environment', name: 'env') ])]) node() { def authToken = "XXXX" stage('Deploy') { deployToOKD(params, authToken) } }
- 30. Lessons learned ● Hide all complexity inside. Provide as much logic as you can in a form of Shared Libraries. ● Documentation & examples are crucial for developers ● If possible - provide standardized pipelines invoked as Shared Library function ● It takes about 1.5 months for 2 people to setup Jenkins properly for the first time
- 31. Single fact you could memorise here Always keep your CI/CD configuration, written as code, near the app, in the same repo. It will give an understanding to everyone in your company on how to build & deploy any app.
- 32. Infrastructure as code: Terraform
- 33. How Terraform is different to Ansible/Chef/Puppet Terraform is not a configuration management tool. It focuses on the higher-level abstraction of the datacenter (or cloud provider), without sacrificing the ability to use configuration management tools to do what they do best: bootstrapping and initializing resources. resource "aws_instance" "web" { ami = "ami-dbc3b9aa" instance_type = "t2.micro" }
- 34. Imperative VS Declarative infrastructure code Declarative “Can I have a cup of coffee on my desk at 9AM on Monday morning?” Imperative “Go to that machine, then get the glass jar, then fill it with water, then put it back in the machine” …you get the idea...
- 35. Why go Declarative? Key challenges this approach solves for us: ● Dealing with “Configuration Drift” / State management ● Idempotency ● Dependency graph management, correct order of operations
- 36. Terraform CI Try now: https://www.runatlantis.io/ https://app.terraform.io/
- 37. Deploy with Terraform What we had: 1. Write TF configs 2. Run TF to create infrastructure 3. Take TF outputs & enter them to Jenkins 4. Deploy app itself with Jenkins What we wanted to have: 1. Write TF configs 2. Deploy app
- 39. Deploy with Terraform !"" vars <-- Environment specific variables # !"" production.tfvars # !"" staging.tfvars !"" main.tf !"" io.tf !"" db.tf !"" etc.tf
- 40. Single fact you could memorize here Try to have as much declarative infrastructure configs as you can, avoid imperative scripts at all cost.
- 41. Learnings & further improvements
- 42. Tests for infrastructure code The more infrastructure code you have - the more bugs you see.
- 43. Chaos monkey We wrote a Lambda that randomly reboots every instance once a day. This simple tweak ensures that: ● Apps you launch can survive instance failure ● Updates to the cluster setup is easy as you’re sure that you won’t harm anyone by killing outdated machines ● Problem resolution can be simpler sometimes. You can always reboot/kill any instance that behaves abnormally as a first action Apply this to control fleet too
- 44. No SSH keys distribution over instances ● If you’re using AWS - simply install SSM agent to your instances and disable SSH daemon. You will be able to use SSH console to perform your administrative actions. ● If you’re not in AWS - you can use Hashicorp Vault. It provides you with SSH backend that allows central management & audit of the login identities.
- 45. Things that work for 3 teams - doesn’t work for 10 Issues # of users
- 46. Key learnings here: ● Operational work grows exponentially the more teams you add ● 1 new tool/approach for devs at a time ● Conduct educational courses for big new things like Docker, AWS, Terraform ● Gain trust within the team you’re challenging with a change first ● Documentation is paramount, start it as early as possible Things that work for 3 teams - doesn’t work for 10
- 47. DevOps on Call & transparent SLAs ● We’ve established “on call” schedule ● Agreed on SLAs & shared then among the teams ● Created chat room & Jira board Result: significant reduction of the distraction level, better productivity, happier teams Even you’re doing good now, you can make it even better with this practice
- 48. Encourage feedback ● Ask for it proactively, show that it's important for you ● Public retrospectives ● Respond to the feedback
- 49. Summing up
- 50. What we’ve achieved ● Immutable infrastructure (done) ○ ECS with immutable data plane ○ Immutable EC2 instances for stateful instances ● Everything as code (done) ○ Infrastructure as code: Terraform, Cloud-init ○ CI/CD as code: Jenkins Now it’s hard to break & easy to repair things as well as easy to track changes.
- 51. Vlad Fedosov Director of R&D @Namecheap, Inc [email protected] Or just scan it: