Dynamic infrastructure for development
- 2. About me
- 3. Motivation ● Startup At the beginning many attempts, chaotic unsystematic code and workflows ● Later on the requirements stabilized It was time to create a scalable developer friendly pipeline ● The road from idea to release was too long ● We needed an easy way to test the applications
- 4. What was at the beginning? ● New branch ● Local build ● (Build or pull the dependencies) ● Deploy it locally (with the dependencies) ● Deploy the database if needed and seed it with dummy data Problem: Not scalable, a more complex service might not have enough resources to run on a local machine, error-prone and hard to maintain.
- 5. Kubernetes to the rescue ● We moved all of our infrastructure to kubernetes ● The infrastructure became more scalable and maintainable ● It allowed us to easily implement zero downtime deployments ● Proper load balancing, health check and automatic restarts out of the box
- 6. Idea: Dynamic infrastructure PoC: ● Branch name identifies the new infrastructure ● Multiple repositories can join the same environment by using the same branch name ● The created infrastructure has a deterministic access point ○ branch.platform.avatao.build ● Every environment gets its own isolated database with proper mock data
- 7. ● Create branch on a repository (e.g.: test) ● A build is started using Google Cloud Build ● New kubernetes namespace is created (called test, derived from the branch) ● Branch is created on the centralized kubernetes manifest repository (test) ● The configuration is changed on this new branch (sed) so that it uses the new namespace (and others like replica number, read-only filesystem etc...) PoC Implementation
- 8. ● hack/deploy-platform.sh (test branch) ○ Copy kubernetes secrets from QA (default namespace in the dev cluster) ○ Generate kubernetes manifest files using envsubst ○ Run managers to provision databases or anything that is needed to deploy the platform ○ Deploy the platform using the QA docker images except for the service that is currently deployed ● Resources like etcd, redis, RabbitMQ are shared amongst namespaces to lower costs and are separated using other techniques like prefixing ● The platform is now available here: test.platform.avatao.build If we use the same branch name on any other repository it will realize that one is already created with the same name so it would just deploy it here. PoC Implementation
- 10. Good as it is? No.
- 11. What's the problem? ● Wasteful, we don't need a new infrastructure for every branch ● The cleaning up is manual and tedious ● The system using branch name as the identifier was easy to implement but it has many flaws ○ I have to rename a branch if I'd like it to join a specific infrastructure ○ What if I accidentally make a branch on my service with an existing infrastructure? ○ One branch can only run in one infrastructure ● If I'd like to change something in the kubernetes manifest files or introduce new microservices I am in trouble ● If anything changes on the QA the synchronization is not easy
- 12. Let's improve it ● Find the right the tool for the goal (CircleCI, Spinnaker) ● Helm instead of envsubst ● Instead of a centralized kubernetes manifest repository, each service is responsible for their own helm chart ● Instead of using the branch as the identifier, develop an internal tool to create infrastructures easily from any branch (it needs to store these information)
- 13. Example #1 ● Developer is working on X and Y service using A and B branches respectively ● It is time to try it out in a feature infrastructure ● Using the internal tool a new feature infra is created where the X and Y services use the A and B branches ● There is a push on either X or Y repository -> using the configuration of the infrastructures we can deploy the app to the proper environment or environments(!).
- 14. Example #2 ● We'd like to deprecate a service or create new ones ● When creating the infrastructure we simply remove the service we'd like to deprecate during creation ● For the new service we choose which repository we'd like to deploy (it needs to have the proper helm chart and spinnaker pipeline ready) ● The infrastructure will be created with the old service removed and the new service deployed
- 15. Garbage collection and maintenance ● If the infrastructure is not needed we can remove it with a click of a button, or it can be automatically deleted after some time of inactivity. ○ Idea: Associate Github PRs to environments and on merged status delete them ● If the QA environment changes we can synchronize it using the internal tool ○ Must not be automatic to prevent breaking change ○ Only update the parts that are not worked on (master branch versions)
- 16. Thank you!