Puppet and AWS is Easy...?
5 likesā¢1,296 views
Puppet and AWS is easy ...... ? This document discusses how Puppet and AWS were used to automate infrastructure at Healthdirect Australia. It describes how manual deployments, inconsistent environments, and a lack of automation were initially problems, and how iterative improvements using Puppet for configuration management and the AWS Puppet module for provisioning infrastructure helped solve these issues and establish continuous delivery of automated testing environments. Key aspects discussed include development best practices, testing frameworks, and integration with deployment tools like Jenkins.
![29
Ā
require => Class['docker']
docker-compose.yml.erb](https://image.slidesharecdn.com/healthdirectpowerpointpuppetcamp-151109215714-lva1-app6892/85/Puppet-and-AWS-is-Easy-29-320.jpg)
Puppet and AWS is Easy...?
- 1. Puppet and AWS is easy ā¦ā¦ ? Cara Crawford and Scott Coulton
- 2. Who are we? DevOps/Infrastructure Manager 2 Ā Cara Crawford ā¢āÆ 8 + years development experience ā¢āÆ 6 + years operational sysadmin experience ā¢āÆ Extensive experience across website solutions in both datacentre and cloud solutions across Linux and Windows environments Ā Ā @cara_p
- 3. Who are we? DevOps Solution Architect 3 Ā Scott Coulton ā¢āÆ 10 + years hosting solutions experience ā¢āÆ Extensive experience in architecture across AWS, Puppet, Linux, Docker and all things open source Sc Ā Ā Ā Ā h#ps://github.com/sco#y-Āāc Ā h#ps:// registry.hub.docker.com /repos/sco#yc/ Ā h#ps://forge.puppetlabs.com/ sco#yc Ā @sco#coulton Ā
- 4. Who are we? 4 Ā Healthdirect Australia manages the following healthcare service: ā¢āÆ Healthdirect nurse helpline and health information ā¢āÆ After hours GP helpline ā¢āÆ Pregnancy, Birth and Baby ā¢āÆ Mindhealthconnect ā¢āÆ National Health Service Directory ā¢āÆ My Aged Care ā¢āÆ Video Call Consulting Ā Ā
- 5. What we were trying to solve August 2013 5 Ā ā¢āÆ Majority of servers located in AWS ā¢āÆ Manual deployments ā¢āÆ Cloning for provisioning ā¢āÆ Inconsistent environments ā¢āÆ No CD ā¢āÆ No CI ā¢āÆ No Infrastructure as code ā¢āÆ No Automated testing ā¢āÆ Very limited automation Ā Ā Deployments would take weeks or months with a high level of developer input onto cloned servers
- 6. How did we solve it 6 Ā First Iteration ā Nov 2013 ā¢āÆ Basic Combination approach of Boto and Python for AWS scripting ā¢āÆ Basic Opensource Puppet for configuration, server management Third Iteration ā Jan 2015 ā¢āÆ AWS Puppet module with cloud formation ā¢āÆ Advanced puppet coding standards with testing ā¢āÆ Container services and schedulers Second Iteration ā Feb 2014 ā¢āÆ Improvements to scripts ā¢āÆ Additional Puppet coding standards including upgrading to Enterprise ā¢āÆ Heading towards 100% Puppet coverage
- 7. Our Environment 7 Ā 5 development environments 3 Puppet masters Dynamically controlling the VPC environments they are responsible for Each environment the access level reducing towards Stage and Prod being automated environments
- 8. So it all looks easy, Right !!!!! 8 Ā
- 9. What have we learnt ? 9 Ā A solid foundation -āÆ Development ā¦ Where to start ??? -āÆ Unit and acceptance testing frameworks -āÆ CD with Puppet Deploying to AWS with Puppet -āÆ Tips and tricks What we are working on now to take Puppet to the next level
- 10. Development, Where to start ? 10 Ā
- 11. If you have never written a module start here !!! 11 Ā h#ps://forge.puppetlabs.com/ Ā
- 12. Start with a solid development foundation !!! 12 Ā h#ps://github.com/mitchellh/vagrant Ā A Ā must Ā have Ā for Ā any Ā Puppet Ā development Ā Ā ā¢āÆ Quick Ā deployments Ā ā¢āÆ A Ā massive Ā community Ā Ā ā¢āÆ Supports Ā any Ā OS Ā you Ā do Ā Ā ā¢āÆ You Ā can Ā deploy Ā a Ā single Ā machine Ā or Ā 100 Ā Ā ā¢āÆ It Ā is Ā as Ā complex Ā or Ā simple Ā as Ā you Ā need Ā it Ā to Ā be Ā Ā ā¢āÆ Its Ā all Ā Ruby Ā Ā Ā
- 13. If you have not developed Vagrant framework , Here are is a good one 13 Ā h#ps://github.com/adrienthebo/oscar Ā
- 14. What we have done with Vagrant 14 Ā ā¢āÆ Re Ā useable Ā VagrantFile Ā with Ā variables Ā ā¢āÆ Yaml Ā box Ā conļ¬guraMon Ā Ā ā¢āÆ MulM-Āānode Ā test Ā environments Ā with Ā both Ā open Ā source Ā and Ā Puppet Ā Enterprise Ā Ā ā¢āÆ Full Ā Hiera Ā integraMon Ā Ā Ā ā¢āÆ We Ā have Ā a Ā full Ā replicaMon Ā of Ā producMon Ā on Ā our Ā laptops Ā Ā Ā
- 15. VagrantFile 15 Ā
- 17. Unit testing is a must !!! 17 Ā ā¢āÆ Picks Ā up Ā human Ā error Ā ā¢āÆ Tests Ā module Ā dependancies Ā ā¢āÆ EssenMal Ā for Ā the Ā CD/CI Ā process Ā Ā Ā Tools for the job h#p://rspec-Āāpuppet.com/ Ā h#ps://github.com/rodjek/rspec-Āāpuppet Ā
- 18. What do tests look like ? 18 Ā A Ā simple Ā example Ā A Ā complex Ā example Ā execute Ā the Ā test Ā ā¢āÆ rake Ā spec_prep Ā && Ā rspec Ā
- 19. But wait what don't test cover ? 19 Ā ā¢āÆ Package Ā versions, Ā If Ā a Ā ļ¬le Ā exists Ā etc Ā So we need acceptance testing ā¢āÆ This Ā where Ā Vagrant Ā comes Ā back Ā in Ā Ā Run the module !!!! Puppetlabs Ā have Ā a Ā very Ā good Ā framework Ā for Ā this Ā h#ps://github.com/puppetlabs/beaker Ā
- 20. So now we have our working modules ā¦ How do we integrate them with CD ? 20 Ā ā¢āÆ r10k Ā h#ps://github.com/puppetlabs/r10k Ā is Ā a Ā good Ā place Ā to Ā start. Ā This Ā will Ā give Ā you Ā a Ā great Ā transport Ā mechanism Ā from Ā source Ā control Ā to Ā the Ā Puppet Ā master. Ā But what if we are using a CD tool (ie Jenkins etc) ?
- 21. Puppet code deployment flow 21 Ā
- 22. So some CD tricks and tips we use 22 Ā Every Ā module Ā has Ā its Ā own Ā repo Ā
- 23. So some CD tricks and tips we use 23 Ā Each Ā pipeline Ā runs Ā diļ¬erent Ā test Ā before Ā it Ā can Ā move Ā to Ā the Ā next Ā
- 24. AWS Deployment Tools 24 Ā ā¦ Ā And Ā because Ā we Ā are Ā not Ā big Ā fans Ā of Ā JSON Ā h#ps://github.com/bazaarvoice/cloudformaMon-Āāruby-Āādsl Ā
- 25. Example of the AWS Puppet module 25 Ā
- 26. Benefits of the AWS Puppet module 26 Ā ā¢āÆ Anyone Ā on Ā the Ā team Ā can Ā develop Ā a Ā new Ā AWS Ā instance, Ā elb Ā etc Ā ā¢āÆ Out Ā of Ā the Ā box Ā we Ā have Ā great Ā unit Ā tesMng Ā coverage Ā ā¢āÆ On Ā every Ā Puppet Ā run Ā reverts Ā the Ā environment Ā to Ā the Ā desired Ā state Ā Number Ā 1 Ā 2p Ā for Ā AWS Ā h#ps://docs.puppetlabs.com/guides/rest_api.html Ā The Ā HTTP Ā API Ā AWS Ā SQS Ā + Ā Puppetlabs Ā HTTP Ā API Ā = Ā Happy Ā engineers Ā Ā Ā
- 27. 27 Ā What we are working on to take Puppet and AWS to the next level
- 28. 28 Ā
- 29. 29 Ā require => Class['docker'] docker-compose.yml.erb
- 30. What we are working on now to take Puppet to the next level 30 Ā Docker Ā scheduling Ā with Ā Puppet Ā Dynamic Ā Hiera Ā backend Ā and Ā service Ā discovery Ā
- 31. So that was easy right ???? 31 Ā