Best Practices for Mission-Critical Jenkins
- 1. Jenkins User Conference New York, May 17 2012 #jenkinsconf Best Practices for a Mission- Critical Jenkins Mike Rooney Consultant/Jenkins Connoisseur http://linkedin.com/in/mcrooney
- 2. Jenkins User Conference New York, May 17 2012 #jenkinsconf Jenkins Uses Genius.com – staging deployment, code reviews, automated branching and merging, monitors Canv.as – continuous deployment, scoring, monitoring, newsletter mailing Conductor – environment creation, staging / prod deployment, selenium monitoring
- 3. Jenkins User Conference New York, May 17 2012 #jenkinsconf Hand-check: How critical is your Jenkins?
- 4. Jenkins User Conference New York, May 17 2012 #jenkinsconf What problems have you faced?
- 5. Jenkins User Conference New York, May 17 2012 #jenkinsconf Problems disk failure / data loss hardware failure / downtime load / latency
- 6. Jenkins User Conference New York, May 17 2012 #jenkinsconf Solution make Jenkins instance trivial to respin – ideally a one-liner that even handles DNS – “create.sh jenkins”
- 7. Jenkins User Conference New York, May 17 2012 #jenkinsconf Persistence $JENKINS_HOME – plugins, users, jobs, builds, configuration
- 8. Jenkins User Conference New York, May 17 2012 #jenkinsconf Persistence git / svn – make $JENKINS_HOME a checkout – have a Jenkins job that commits daily – examples: http://jenkins- ci.org/content/keeping-your-configuration- and-data-subversion
- 9. Jenkins User Conference New York, May 17 2012 #jenkinsconf Persistence EBS on AWS – put $JENKINS_HOME on an EBS volume – snapshot nightly via a Jenkins job – trivial to attach to a new host, restore snapshot a NAS + RAID / backups works similarly
- 10. Jenkins User Conference New York, May 17 2012 #jenkinsconf Environment Jenkins is more than $JENKINS_HOME – specific Jenkins .war / .deb / .rpm version – startup options – dependent packages: git, ruby gems, pip – ssh keys, m2 settings – swap, tmpfs, system configuration
- 11. Jenkins User Conference New York, May 17 2012 #jenkinsconf Environment configuration management:Puppet/Chef* * https://wiki.jenkins-ci.org/display/JENKINS/Puppet
- 12. Jenkins User Conference New York, May 17 2012 #jenkinsconf Environment standalone – puppet apply path/to/your/manifest.pp puppetmaster – set up /etc/puppet.conf, run puppet agent
- 13. Jenkins User Conference New York, May 17 2012 #jenkinsconf Putting it Together have manifest handle $JENKINS_HOME – clone git repo, mount EBS volume, etc
- 14. Jenkins User Conference New York, May 17 2012 #jenkinsconf Putting it Together…on AWS upload manifests to S3 on check-in – a Jenkins SCM job using S3 plugin use cloud-init to install puppet, download manifests, and run puppet – a custom AMI with an rc.local script also works when it dies: “create.sh jenkins” – ec2-launch-instance config user-data
- 15. Jenkins User Conference New York, May 17 2012 #jenkinsconf Monitoring … but how do you know when it’s down? check out services like Pingdom – notifies you when a URL does give HTTP 200 OK
- 16. Jenkins User Conference New York, May 17 2012 #jenkinsconf Going further: Elastic Beanstalk handles provisioning simply from a .war pros – just give it a war – automatically replaces unhealthy instances – behind a load-balancer (consistent URL) – normally hard AWS changes like AMI, Security Groups, or Key Pairs are now trivial to make cons – behind a load-balancer (cost overhead) – no UI option (yet) for controlling AZ – no great way to pass data to instances for puppet – locked in to Amazon Linux AMI (CentOS)
- 17. Jenkins User Conference New York, May 17 2012 #jenkinsconf Going further: Elastic Beanstalk set min/max instances to 1 – ignore scaling triggers, irrelevant in this case use beanstalk CLI to set desired AZ (if EBS) – https://forums.aws.amazon.com/thread.jspa?t hreadID=61409 puppet – use a custom AMI that specifically runs Jenkins manifests – but this requires a specific AMI for each Beanstalk application. – let’s get creative…
- 18. Jenkins User Conference New York, May 17 2012 #jenkinsconf Going further: Elastic Beanstalk passing data to instances PARAM1..5 meant as args to .war end up in /etc/sysconfig/tomcat7 JAVA_OPTS parse out and: – puppet apply –certname=$PARSED_ROLE
- 19. Jenkins User Conference New York, May 17 2012 #jenkinsconf Questions?
- 20. Jenkins User Conference New York, May 17 2012 #jenkinsconf High Availability Artifacts protect: artifacts, reports, userContent from: – planned downtime: Jenkins restarts/upgrades, server upgrades – unplanned downtime: software/hardware failure – unresponsive Jenkins: very high load
- 21. Jenkins User Conference New York, May 17 2012 #jenkinsconf High(er) Availability Artifacts easy mode: – put Jenkins behind nginx/apache, shadow userContent and relevant directories – still available during Jenkins restarts, or very high Jenkins load/latency – not safe from server downtime
- 22. Jenkins User Conference New York, May 17 2012 #jenkinsconf High Availability Artifacts advanced mode: S3 – 99.99% availability, 99.999999999% durability* • if you store 10K objects, expect to lose one every 10 million years – use Jenkins S3 plugin to upload artifacts to S3 * http://aws.amazon.com/s3/faqs
- 23. Jenkins User Conference New York, May 17 2012 #jenkinsconf Fault-tolerant Jobs design with possible downtime in mind – SCM triggering is great, but keep polling too
- 24. Jenkins User Conference New York, May 17 2012 #jenkinsconf Fault-tolerant Jobs */15 * * * * – BAD: update users where join_time < 15m ago – GOOD: update users where id > last_id_updated
- 25. Jenkins User Conference New York, May 17 2012 #jenkinsconf Error handling for non-critical jobs, use email / IM post- build notifiers – but be careful of creating too much noise, people will ignore or filter it out for critical jobs, integrate Jenkins with a service like PagerDuty – Jenkins emails [email protected] – PagerDuty texts / calls the people on-call until resolved – a failing build will wake you up at 4AM
- 26. Jenkins User Conference New York, May 17 2012 #jenkinsconf Questions?
- 27. Jenkins User Conference New York, May 17 2012 #jenkinsconf Security: Authentication read-only matrix-based HTTP basic auth
- 28. Jenkins User Conference New York, May 17 2012 #jenkinsconf Security: Authentication but what about traffic sniffing?
- 29. Jenkins User Conference New York, May 17 2012 #jenkinsconf Security: HTTPS throw nginx/apache in front of Jenkins – proxy mode – ssl (self-signed or just buy one)
- 30. Jenkins User Conference New York, May 17 2012 #jenkinsconf Security: Authorization use project-based matrix authentication give anonymous/authenticated readonly use it if you’ve got it: LDAP, Active Directory, UNIX Jenkin’s own database also works fine ensure each user has their own account – each build will have an audit trail
- 31. Jenkins User Conference New York, May 17 2012 #jenkinsconf Security: Authorization (AWS) when interfacing with AWS API/CLI, use IAM so Jenkins can only access what it needs
- 32. Jenkins User Conference New York, May 17 2012 #jenkinsconf Questions?
- 33. Jenkins User Conference New York, May 17 2012 #jenkinsconf Thank You To Our Sponsors Platinum Sponsor Gold Sponsors Silver Sponsors Bronze Sponsors