Continuously-Integrated Puppet in a Dynamic Environment
17 likes•12,110 views
The document discusses the implementation of a masterless Puppet configuration in a dynamic environment, detailing the advantages and challenges of using Amazon Web Services (AWS) for infrastructure management. It outlines the workflow involving tools like CloudFormation, Pulp, and Jenkins for managing deployments, configuration as code, and continuous integration, while addressing issues with traditional Puppet deployment methods that assume servers are 'pets.' The author proposes solutions to these challenges, emphasizing efficiency, reusability, and improved operational control.
Continuously-Integrated Puppet in a Dynamic Environment
- 1. CONTINUOUSLY INTEGRATEDPUPPETIN A DYNAMIC ENVIRONMENT SAM BASHTON, BASHTON LTD
- 3. CONTINUOUSLY INTEGRATEDPUPPETIN A DYNAMIC ENVIRONMENT MASTERLESSPUPPET:WHY ANDHOW SAM BASHTON, BASHTON LTD
- 4. ABOUTME Linux guy since Slackware, floppy disks and root + boot Using Puppet since 2007 Run a company in Manchester, North West England We provide outsourced ops for other companies
- 5. OUR FULLYMANAGED ENVIRONMENTS Primarily transactional websites (e-commerce) Majority (70%+) on Amazon Web Services (AWS) Majority using CentOS
- 6. HOWWEWORK Simple is better than complex Complexity is worth adding only if it provides obvious functional benefits Re-usability Resilience
- 7. WHY DID WE PICK AWS? Featureset and toolset massively in advance of any other cloud provider, public or private #1 customer reason for switching to AWS? The ability to scale on demand
- 9. TOOLSWEUSEFOR BUILDINGANDMANAGING Do one thing and do it well CloudFormation - Amazon tool to manage infrastructure Puppet - Manage system configuration Pulp - centralised repository, manages package revisions Jenkins
- 10. HOWWEUSEPUPPET No Puppetmaster Puppet manifests, hieradata and modules distributed to all machines via RPM All machines boot with a common, blank image and get configured at first boot
- 11. WHAT'SWRONGWITH MASTER BASEDPUPPET? Pets vs Cattle Puppet designed for a world of servers as pets We do not live in that world
- 12. PUPPETDESIGNEDFOR PETS Many assumptions in Puppet presume that your servers are pets Some of these work against us when managing a herd
- 13. MANUALCERTIFICATE SIGNING Clearly unsuitable when machines are automatically provisioned
- 14. POTENTIAL WORKAROUNDS: Autosign Use/write another automated certificate generation mechanism Possibly tied in with autoscaling
- 15. NO MECHANISMFOR CLEANINGOLDHOSTS Likely to have host-names reused, causing machines to fail to configure Puppetmaster will fill with certificates for machines that ran for a few hours and went away again
- 16. POTENTIAL WORKAROUNDS: Use UUID certificates Agree not to look in the certificate directory Write mechanism for cleaning up old certificates
- 17. HOSTSCONFIGUREDBASED ON HOSTNAME Our machines have names like ip-172-26-5-123 How does Puppet know what type of machine this is?
- 18. POTENTIAL WORKAROUNDS Use an external node classifier Use some mechanism for giving a better hostname, eg web-172-26-5-123 and use regex for node names
- 19. PUPPETMASTER ISASINGLE POINTOFFAILURE If the Puppetmaster fails, we can no longer autoscale up In particular, this could be a problem if there is availability zone failure
- 21. WORKAROUNDRECAP Use/write alternative certificate management software Write an external node classifier / mechanism for setting hostname appropriately Cluster multiple Puppetmasters
- 22. WHATWEDIDINSTEAD Decided using a Puppetmaster was trying to fit a square peg into a round hole Instead, decided to run Puppet without a master
- 24. DISTRIBUTINGMANIFESTS Use RPM Distribute full set of manifests/modules to each machine Apply only the manifest relevant to that machine
- 25. PACKINGPUPPET MANIFESTSIN RPM Build an RPM containing everything under /etc/puppet Make files readable only by root
- 26. APPLYPUPPETMANIFESTS Have an RPM %postinstcommand apply the Puppet config This isn't as straightforward as running the puppet applyfrom %postinst Puppet needs to install packages via yum, but yum is running installing the Puppet package Instead, we work around with a dirty hack: have the %postinstcreate an atscript which checks if yum has finished and then runs the puppet apply
- 27. RPMINSTALLATION AND MANAGEMENT How do we get these RPMs on our machines?
- 28. PULP We were already using Pulp Provides yum repository management Used for managing security updates and deploying application code http://pulpproject.org/
- 29. WHATISPULP Repository manager Allows us to easily audit what packages and versions are installed where Allows us to push package installations Uses qpid message queue Has concept of 'content distrubtion servers' for easy replication and clustering
- 30. HOWWEUSEPULP Puppet contains details of what packages should be installed Pulp manages which version of the package should be installed Pulp allows us to clone repos and copy packages between them for easy qa->stage->live environment management
- 31. DEPLOYING CONFIGURATION ASCODE Allows us to reuse our existing code deployment infrastructure Manage configuration deployment from Jenkins
- 32. HOWWEDEPLOYCODE Everything managed via the Jenkins continuous integration server Jenkins uses Pulp to install code on remote machines
- 33. DETAILSON HOWWE DEPLOYCODE Jenkins fetches code from source control (git) An RPM is built Tests are run If tests pass, the RPM is added to the relevant Pulp repository RPM installed on the target machine(s)
- 34. DEPLOYMENTLIFE-CYCLE Jenkins also manages deployment life-cycle RPMs are installed on staging Promoted Builds plugin then used to install the same RPMs on live once testing is complete
- 35. PUPPETDEPLOYMENT PROCESS Puppet manifests are checked into git Lint tests via Jenkins pulls in modules with librarian-puppet, then builds an RPM Deployment to test environments, functional tests for wider code-base run Jenkins Warnings plugin
- 36. PUTTINGITINTO PRODUCTION Once suitable tests (automated and manual) have been carried out, we promote Puppet config into production We use the Jenkins 'Promoted Builds' plugin for this
- 38. EXCEPT.. How does a machine get from a bare image to the state where we can push packages to it from Pulp? How does a machine know what type of machine it is? How do we find other resources, eg database hostname?
- 39. CLOUDFORMATION Amazon tool for specifying infrastructure Everything* we provision inside AWS is provisioned via CloudFormation JSON templates * Everything except for the things Amazon doesn't expose via CloudFormation..
- 40. CLOUD-INIT Works with multiple cloud types Sorts out things like SSH keys, allows us to configure host names Also allows us to provide a bash script to run on startup
- 41. PROVISIONINGABARE INSTANCE cloud-init automatically manually adds the pulp repo which contains Pulp, Puppet and our Puppet manifests/modules Installs appropriate RPMs Puppet runs, subscribing the machine to the relevant Pulp repos, and installing packages in the usual Puppet way
- 42. HOWDOESITKNOWWHAT TYPEOFMACHINEITIS? We tell it! Use an environmental variable $HOSTTYPE Simply run puppet apply --modulepath=/etc/puppet/modules ${HOSTTYPE}.pp
- 43. EXTRAFACTS Custom facter facts Also specified in an environmental variable Data comes from within the CloudFormation template On our list of things to look at: FACTER_HOSTENVIRONMENT=live FACTER_STACKNAME=customer-web-live https://github.com/fanduel/hiera-cloudformation
- 44. OTHER RESOURCES We either: Provide details as a facter fact `FACTER_DBHOST=xyz Also use this approach to limit distribution of secure details, eg DB passwords Discover via the EC2 API Eg Varnish servers discover web backends by calling API and finding hosts tagged appropriately
- 45. FREEWINS!
- 46. FREEWINS! Greater control over the timing of Puppet runs Improved visibility - for ops and devs Configuration changes now have to be deployed to testing/staging first
- 47. MOREFREEWINS! Puppet configs now have a version Easy to find config version on the machine itself Config changelogs accessible on every machine (Git changelog added to RPM)
- 48. THEDOWNSIDES Puppet manifests and modules on all machines Potentially a security issue? Mitigated by CloudFormation holding most sensitive data
- 49. ALTERNATIVE IMPLEMENTATIONS Don't want to use Pulp? Could do basically the same thing with yum s3 plugin Use mcollective to push package updates https://github.com/jbraeuer/yum-s3-plugin
- 50. FUTUREIMPROVEMENTS Build AMIs using Packer instead of configuring at boot time Decrease time to autoscale Would probably still need to run Puppet at first boot to configure machine specific settings
- 51. QUESTIONS? COMMENTS? Sam Bashton [email protected] Twitter: @bashtoni (Psst.. )http://www.bashton.com/jobs/