Scaling to 150,000 Builds a Month... and Beyond
4 likes1,999 views
Peter Leschev discusses Atlassian's build engineering practices, highlighting the infrastructure and tools used to support scaling and reliability in CI/CD processes. With a significant increase in builds, the team has adopted practices such as infrastructure as code and automation tools like Puppet and Vagrant to enhance efficiency. The presentation emphasizes the importance of continuous improvement and collaboration within the team to navigate the challenges of managing a large build environment.
Scaling to 150,000 Builds a Month... and Beyond
- 1. PETER LESCHEV • TEAM LEAD • ATLASSIAN • @PETERLESCHEV Build Engineering @ Atlassian: Scaling to 150k builds per month & beyond Summit 2015
- 3. Build platform & services used internally within Atlassian to build, test & deliver software
- 4. Developers expect a reliable infrastructure & fast CI feedback
- 5. • 12 Bamboo Servers • maven.atlassian.com / 9 Nexus instances / 9 TB • 7 Nexus proxies for internal traffic • Monitoring • opsview, graphite, statsd, newrelic, datadog Build Engineering today @ Atlassian • 1200 build agents on EC2 • include SCM clients, JDKs, JVM build tools, databases, headless browser testing, Python builds, NodeJS, installers & more • Maintain 20 AMIs of various build configurations
- 6. 4 years ago: Builds per month 21k
- 7. Last month: Builds per month 225k
- 8. Build Engineering @ Atlassian
- 10. JIRA alone has Automated tests 49k
- 12. 3 stories of gaining maturity to handle Atlassian growth
- 15. Individual Engineers Information Silos Fault investigation, requests for advice, unplanned work Little project work Very interrupt driven Duplication of effort Limited to customer driven changes
- 16. Disturbed role Knowledge Transfer When switching between project / disturbed roles is difficult More project work Non-disturbed can focus on larger tasks Context switching Reduction in duplication of effort, promotes collaboration within the team 2 week rotation
- 17. Team expands Infra Engineers Developers Build Engineers
- 18. Disturbed for Dev & Infra Too interrupt driven To encourage knowledge transfer between infra & dev Staggered changeovers Minimising disruption due to context switching Disturbed pairing Couldn’t handle smaller customer raised requests & interrupt driven work
- 22. 1. Measure the pain 2. Continuous Improvement
- 23. Technical Debt
- 24. Technical Debt
- 25. Contact Rate + Confluence Questions + Hipchat queries Customer JIRA issues Number of Developers ( ) ÷ =
- 26. Contact Rate
- 28. Rebranding Maintenance Disturbed Removing the negative attitude towards the old role within the team
- 30. How do we avoid this in the future? PETER LESCHEV “ ”
- 31. Fix it now, fix it for the future
- 32. Self service
- 34. Self Service Maven Self Help Tool
- 36. Infrastructure as Code = Puppet + SCM ?
- 37. 4 years ago… Started using Puppet Manually maintained snow flakes
- 41. • Coding on Puppet Master • Culture of manually modifying production - Configuration Drift • Impact on Builds Using Staging for Development puppetmaster build agents staging puppet environment
- 43. Rolling out to staging Rolling out to production Broken build agents Developing locally
- 45. But it works on my machine EVERY DEVELOPER “ ”
- 46. Continuous Integration ‘From scratch’ provisioning Confidence that you can rebuild in disaster
- 47. The Pets: you give nice names, you stroke them, and when they get ill, you nurse them back to health, taking a long time over it. “ ” The Cattle: you give them numbers. When they get ill, you shoot them TIM BELL, CERN
- 51. Manual Puppet Rollouts git clone librarian-puppet install symlink update on puppet master
- 52. Bamboo Deployments Available agents Available agents Destination server TASK 1 TASK 2 Task list 1.3 Build results (Artifacts) Release Environments Repository Build artifacts Release 1.3 Build results (Artifacts) Release 1.0 1.3 1.3 Build results (Artifacts) Release Environments Production Development 1.0 1.3 1.3 1.3 Build results (Artifacts) Release Environments Production Development 1.0 1.31.3 Production Development 1.31.3 Development Repository Build artifacts Release How artifacts work 1.3 1.0 1.31.3 Production Development 1.31.3 Development Release notes Repository Build artifacts Release staging production 1.3 Build results (Artifacts) Release Environments Repository Build artifacts Release 1.3 d results rtifacts) Release Environments sitory Build artifacts Release 1.3 Build results (Artifacts) Release Environments Repository Build artifacts Release build • git clone • librarian-puppet • to specific environments • scp to puppet master & symlink update test deploy • ‘delta’ & ‘from scratch’ vagrant provisions 1.3 Build artifacts Release build & test AMIs • Generated using Packer • AMIs on Bamboo Servers updated deploy AMIs
- 53. Puppet Build, Test & Deploy Pipeline
- 54. Puppet Build, Test & Deploy Pipeline
- 55. ‘open prs’ Bot
- 56. Less human effort through automation = Increased frequency & reliability of releases
- 60. At scale is hard
- 62. Build Plans 3500
- 64. Bamboo is great, but hard to manage at scale
- 67. Plan Templates Checked into SCM Bamboo Plugin: Reusable snippets changes can be code reviewed Export plans for backup, or move to another Bamboo instance easily Bulk changes Export existing plans Update 100s of job requirements with a single commit
- 68. Pushing Bamboo to its limits
- 69. Agent Smith Wallboard Bamboo Plugin: Trend data sent to Graphite https://marketplace.atlassian.com/plugins/com.atlassian.bamboo.plugin.agent-smith-wallboard
- 70. Add metrics, then alert on them
- 71. Bamboo Monitoring Plugin Metrics to graphite Bamboo Plugin: Bamboo Health ActiveMQ, Database connections, Tomcat, JVM Memory usage. Background thread workers. Number of plans / plan branches, plans / plan branches for deletion. https://marketplace.atlassian.com/plugins/com.atlassian.bamboo.plugin.bamboo-monitoring-plugin
- 73. When a Bamboo Server starts misbehaving…
- 74. Infrastructure differences? Is it Bamboo Configuration? Is it a Bamboo Plugin? Is it Bamboo the product? How is it being used?
- 75. Infrastructure consistency of Bamboo Servers is key
- 76. Bamboo Puppet provider + REST API for Administration Bamboo Puppet Provider REST calls https://forge.puppetlabs.com/atlassian/bamboo_rest
- 78. Bamboo Plugins ‘Continuous Plugin Deployment’ Task This text box is not intended to contain a bunch of copy. 1-click upgrades of Available agents Available agents Destination server TASK 1 TASK 2 Task list Available agentsTask list Task list Production Available agents Destination server TASK 1 TASK 2 Task list How artifacts work 1.3 1.0 1.31.3 Produc Developm 1.31.3 Development Release notes Repository Build artifacts Release How artifacts work 1.3 Build results (Artifacts) Release Environments 1.0 1.31.3 Production Development 1.31.3 Development Release notes Repository Build artifacts Release How artifacts work 1.3 1.0 1.31.3 Production Development 1.31.3 Development Artifacts Ve Test & Build JIRA issue Commit TriggerCode Release notes Repository Build artifacts Release All Bamboo Servers How artifacts work 1.3 Build results (Artifacts) Release Environments 1.0 1.31.3 Production Development 1.31.3 Development Repository Build artifacts Release build Deploy How artifacts work 1.3 1.0 1.31.3 Production Developmen 1.31.3 Development Repository Build artifacts Release build & test AMIs Build https://marketplace.atlassian.com/plugins/com.atlassian.bamboo.plugins.deploy.continuous-plugin-deployment
- 79. Bamboo Servers 1-click upgrades of Using scp / ssh & puppet Available agents Available agents Destination server TASK 1 TASK 2 TASK 1 TASK 2 Task list Available agents TASK 1 TASK 2 Task list Task list Production Available agents Destination server TASK 1 TASK 2 Task list How artifacts work 1.3 Build results (Artifacts) Release Environments De 1.0 1.31.3 Production Development 1.31.3 Development Repository Build artifacts Release How artifacts work 1.3 1.3 1.3 Build results (Artifacts) Release Environments Development 1.0 1.31.3 Production Development 1.31.3 Development Release notes Repository Build artifacts Release How artifacts work 1.3 1.0 1.31.3 Production Development 1.31.3 Development Artifacts Versio Test &JIRA Release notes Repository Build artifacts Release How artifacts work 1 1.3 1.3 Build results (Artifacts) Release Environments Development 1.0 1.31.3 Production Development 1.31.3 Development Repository Build artifacts Release Upgrade Bamboo 1.3 1.0 1.31.3 Production Development 1.31.3 Development Repository Build artifacts Release Build Bamboo 1.3 Build results (Artifacts) Release Environm D Repository Build artifacts 1.0 1.3 1.3 1.3 Build results (Artifacts) Release Environments Production Development Repository Build artifacts Release jira-bamboo servicedesk-bamboo
- 80. Infrastructure differences? Is it Bamboo Configuration? Is it a Bamboo Plugin? Is it Bamboo the product? How is it being used?
- 83. We’ve matured to handle the growth of Atlassian
- 84. Thank you! PETER LESCHEV • TEAM LEAD • ATLASSIAN • @PETERLESCHEV