© 2019 CloudBees, Inc. All Rights Reserved.© 2019 CloudBees, Inc. All Rights Reserved.
CI/CD Best Practices
for your DevOps Journey
November 4, 2019
© 2019 CloudBees, Inc. All Rights Reserved.
Today’s Speakers
CloudBees: Brian Dawson
DevOps Evangelist
@brianvdawson
DevOps.com: Mitchell Ashley
Director of Research
Mitchell Ashley
CloudBees: Doug Tidwell
Technical Marketing Director
Doug3000
© 2019 CloudBees, Inc. All Rights Reserved.
Webinar Overview
Agenda
● Beyond DevOps: Software Delivery Management
● Summary of 10 CI/CD best practices
● “Double-Click” on key best practices
● Breaks for Discussion
● Questions and Answers
© 2019 CloudBees, Inc. All Rights Reserved.
DevOps Elite Performers
46x
more frequent code deploys
i.e. multiple times per day vs. once a week or less
2406x
faster time to recover from downtime
i.e. less than an hour vs. weeks
2555x
faster lead-time from commit to deploy
i.e. less than an hour vs. more than a week
1/7th
as likely that changes will fail
i.e. fail 0-15% of the time vs. 46-60%
From https://devops-research.com/
© 2019 CloudBees, Inc. All Rights Reserved.© 2019 CloudBees, Inc. All Rights Reserved.
But...
Things aren’t perfect.
We still have...
● Disparate, stand-alone software tools
● No common language, data or process sets
● No clear way to ensure we deliver the right thing
Disconnected and Fragmented teams, tools and process
© 2019 CloudBees, Inc. All Rights Reserved.
Executive,
Finance
CPO, PM,
Design
Software
Development and Delivery
IT and Engineering
Dev QA Ops
Marketing,
Docs,
Training
Sales
Support,
Services
Software Delivery
It Takes a Village to Truly Deliver Value
© 2019 CloudBees, Inc. All Rights Reserved.
Research Design Build Test Deploy Release Observe Support
Sales/MktgSecurity/OpsDev/QA
CPO/Product
Mgmt/ Design
Support/ServicesExec/ Finance
Identify/Plan
UNIVERSAL INSIGHTS
COMMON DATA
Jira
Confluence
Twistlock Jenkins XGitHub
SonarQubeArtifactory Rollout
Twistlock Flow
Splunk
Existing Enterprise Tools
Introducing Software Delivery Management
All functions collaborating
COMMON CONNECTED PROCESS
Slack
Salesforce
© 2019 CloudBees, Inc. All Rights Reserved.
We have to Start with CI/CD and DevOps
Technical Concerns
Software Delivery Management
Business Concerns
Code, Build
Release
Value and Impact
Strategy and Revenue
Research Code Build Test Deploy Release Observe SupportIdentify/Plan
CI
CD
DevOps
Test
Full Software Value Lifecycle
Sell
© 2019 CloudBees, Inc. All Rights Reserved.
But the reality is...
Far fewer people are truly practicing CI/CD than is widely
reported!
https://www.cloudbees.com/jenkins/devops-and-jenkins-community-survey-2019
© 2019 CloudBees, Inc. All Rights Reserved.
The Path to the Future
As the industry positions itself to build on DevOps practices
with a Software Delivery Management strategy…
...it’s more important than ever that we implement CI/CD best
practices, and prepare for the future.
© 2019 CloudBees, Inc. All Rights Reserved.
10 CI/CD Best Practices Summary
1. Track work items
2. Use source code management
3. Tags, not Branches
4. Automate the builds
5. Stop the line when the build breaks
6. Validate and test
7. Deploy
8. Improve incrementally
9. Collaboration
10.Create a true DevOps culture
© 2019 CloudBees, Inc. All Rights Reserved.© 2019 CloudBees, Inc. All Rights Reserved.
Best Practice #1:
Track Work Items
© 2019 CloudBees, Inc. All Rights Reserved.
Track Work Items
Use a system that lets you track everything related to your
release:
● User Stories
● Bugs
● Infrastructure and environment
This is useful for CI and CD, but is absolutely critical as you
move to DevOps and eventually Software Delivery
Management (SDM).
© 2019 CloudBees, Inc. All Rights Reserved.© 2019 CloudBees, Inc. All Rights Reserved.
Best Practice #2:
Use Source Code Management
© 2019 CloudBees, Inc. All Rights Reserved.
Use Source Code Management (SCM)
Use an SCM tool to track all the changes made for a
release.
A decentralized version control system such as git
A centralized version control system such as SVN
Must consider support for new and legacy systems
© 2019 CloudBees, Inc. All Rights Reserved.
Use Source Code Management
Encourage commit best practices
If culture is to not commit frequently, it won’t matter. If
a developer waits three weeks to commit or branches
off for three weeks, they have delayed the integration
and broken the principles
● Commit frequently
● Minimize duration of branches
© 2019 CloudBees, Inc. All Rights Reserved.
Use Source Code Management
Consider how triggers that launch builds are initiated.
● Trigger on every commit
● Trigger on developer branch
● Trigger on pull request
● Trigger on merge
© 2019 CloudBees, Inc. All Rights Reserved.© 2019 CloudBees, Inc. All Rights Reserved.
Best Practice #3:
Tags, not Branches
© 2019 CloudBees, Inc. All Rights Reserved.
Tags, not Branches
Commit or Merge to the trunk frequently, or even better,
always commit to the trunk/master
Developers often branch and maintain changes off of a
trunk to manage releases.
But branching creates complexity that prevents
everyone from working with a single source of truth, and
introduces maintenance overhead
● Use tags to manage releases
● Implement functionality incrementally
● Use feature flags
© 2019 CloudBees, Inc. All Rights Reserved.© 2019 CloudBees, Inc. All Rights Reserved.
Best Practice #4:
Automate the Builds
© 2019 CloudBees, Inc. All Rights Reserved.
Automate the Builds
Centralize and automate the entire build process
● Compiling the source code
● Packaging the compiled code
● Container images
The build process should run for every commit.
© 2019 CloudBees, Inc. All Rights Reserved.
Automate the Builds
The build should run as quickly as possible
The builds should take no more than 5-10 minutes.
Commit should be blocked during build, to support
“stop-the-line” culture. If builds are long developers
defer commits.
● Build
● Code scans, unit test
● Functional tests if possible
Promote the build to extended CI/CD loop for longer
activities
© 2019 CloudBees, Inc. All Rights Reserved.© 2019 CloudBees, Inc. All Rights Reserved.
Best Practice #6:
Validate and Test
© 2019 CloudBees, Inc. All Rights Reserved.
Validate and Test
Organizations that don’t validate every build are not
practicing CI.
Organizations that do not continuously test are not
practicing CD.
● Manage tests like code or as code
● Fail if the build if test thresholds aren’t met
● Duplicate the production environment as much as
possible
© 2019 CloudBees, Inc. All Rights Reserved.© 2019 CloudBees, Inc. All Rights Reserved.
Best Practice #10:
Create a True DevOps Culture
© 2019 CloudBees, Inc. All Rights Reserved.
Create a True DevOps Culture
This may be the most difficult practice to implement, but
no DevOps transformation is possible without it.
Starting a successful DevOps journey requires
significant culture change, including:
● Executive support
● Training
● Funding
● A new mindset for everyone: the CIO/CTO,
release managers, engineering managers, the ops
team, and the dev team.
But..start, learn, and improve
© 2019 CloudBees, Inc. All Rights Reserved.© 2019 CloudBees, Inc. All Rights Reserved.
Thank You
Q & A

CI/CD Best Practices for Your DevOps Journey

  • 1.
    © 2019 CloudBees,Inc. All Rights Reserved.© 2019 CloudBees, Inc. All Rights Reserved. CI/CD Best Practices for your DevOps Journey November 4, 2019
  • 2.
    © 2019 CloudBees,Inc. All Rights Reserved. Today’s Speakers CloudBees: Brian Dawson DevOps Evangelist @brianvdawson DevOps.com: Mitchell Ashley Director of Research Mitchell Ashley CloudBees: Doug Tidwell Technical Marketing Director Doug3000
  • 3.
    © 2019 CloudBees,Inc. All Rights Reserved. Webinar Overview Agenda ● Beyond DevOps: Software Delivery Management ● Summary of 10 CI/CD best practices ● “Double-Click” on key best practices ● Breaks for Discussion ● Questions and Answers
  • 4.
    © 2019 CloudBees,Inc. All Rights Reserved. DevOps Elite Performers 46x more frequent code deploys i.e. multiple times per day vs. once a week or less 2406x faster time to recover from downtime i.e. less than an hour vs. weeks 2555x faster lead-time from commit to deploy i.e. less than an hour vs. more than a week 1/7th as likely that changes will fail i.e. fail 0-15% of the time vs. 46-60% From https://devops-research.com/
  • 5.
    © 2019 CloudBees,Inc. All Rights Reserved.© 2019 CloudBees, Inc. All Rights Reserved. But... Things aren’t perfect.
  • 6.
    We still have... ●Disparate, stand-alone software tools ● No common language, data or process sets ● No clear way to ensure we deliver the right thing Disconnected and Fragmented teams, tools and process
  • 7.
    © 2019 CloudBees,Inc. All Rights Reserved. Executive, Finance CPO, PM, Design Software Development and Delivery IT and Engineering Dev QA Ops Marketing, Docs, Training Sales Support, Services Software Delivery It Takes a Village to Truly Deliver Value
  • 8.
    © 2019 CloudBees,Inc. All Rights Reserved. Research Design Build Test Deploy Release Observe Support Sales/MktgSecurity/OpsDev/QA CPO/Product Mgmt/ Design Support/ServicesExec/ Finance Identify/Plan UNIVERSAL INSIGHTS COMMON DATA Jira Confluence Twistlock Jenkins XGitHub SonarQubeArtifactory Rollout Twistlock Flow Splunk Existing Enterprise Tools Introducing Software Delivery Management All functions collaborating COMMON CONNECTED PROCESS Slack Salesforce
  • 9.
    © 2019 CloudBees,Inc. All Rights Reserved. We have to Start with CI/CD and DevOps Technical Concerns Software Delivery Management Business Concerns Code, Build Release Value and Impact Strategy and Revenue Research Code Build Test Deploy Release Observe SupportIdentify/Plan CI CD DevOps Test Full Software Value Lifecycle Sell
  • 10.
    © 2019 CloudBees,Inc. All Rights Reserved. But the reality is... Far fewer people are truly practicing CI/CD than is widely reported! https://www.cloudbees.com/jenkins/devops-and-jenkins-community-survey-2019
  • 11.
    © 2019 CloudBees,Inc. All Rights Reserved. The Path to the Future As the industry positions itself to build on DevOps practices with a Software Delivery Management strategy… ...it’s more important than ever that we implement CI/CD best practices, and prepare for the future.
  • 12.
    © 2019 CloudBees,Inc. All Rights Reserved. 10 CI/CD Best Practices Summary 1. Track work items 2. Use source code management 3. Tags, not Branches 4. Automate the builds 5. Stop the line when the build breaks 6. Validate and test 7. Deploy 8. Improve incrementally 9. Collaboration 10.Create a true DevOps culture
  • 13.
    © 2019 CloudBees,Inc. All Rights Reserved.© 2019 CloudBees, Inc. All Rights Reserved. Best Practice #1: Track Work Items
  • 14.
    © 2019 CloudBees,Inc. All Rights Reserved. Track Work Items Use a system that lets you track everything related to your release: ● User Stories ● Bugs ● Infrastructure and environment This is useful for CI and CD, but is absolutely critical as you move to DevOps and eventually Software Delivery Management (SDM).
  • 15.
    © 2019 CloudBees,Inc. All Rights Reserved.© 2019 CloudBees, Inc. All Rights Reserved. Best Practice #2: Use Source Code Management
  • 16.
    © 2019 CloudBees,Inc. All Rights Reserved. Use Source Code Management (SCM) Use an SCM tool to track all the changes made for a release. A decentralized version control system such as git A centralized version control system such as SVN Must consider support for new and legacy systems
  • 17.
    © 2019 CloudBees,Inc. All Rights Reserved. Use Source Code Management Encourage commit best practices If culture is to not commit frequently, it won’t matter. If a developer waits three weeks to commit or branches off for three weeks, they have delayed the integration and broken the principles ● Commit frequently ● Minimize duration of branches
  • 18.
    © 2019 CloudBees,Inc. All Rights Reserved. Use Source Code Management Consider how triggers that launch builds are initiated. ● Trigger on every commit ● Trigger on developer branch ● Trigger on pull request ● Trigger on merge
  • 19.
    © 2019 CloudBees,Inc. All Rights Reserved.© 2019 CloudBees, Inc. All Rights Reserved. Best Practice #3: Tags, not Branches
  • 20.
    © 2019 CloudBees,Inc. All Rights Reserved. Tags, not Branches Commit or Merge to the trunk frequently, or even better, always commit to the trunk/master Developers often branch and maintain changes off of a trunk to manage releases. But branching creates complexity that prevents everyone from working with a single source of truth, and introduces maintenance overhead ● Use tags to manage releases ● Implement functionality incrementally ● Use feature flags
  • 21.
    © 2019 CloudBees,Inc. All Rights Reserved.© 2019 CloudBees, Inc. All Rights Reserved. Best Practice #4: Automate the Builds
  • 22.
    © 2019 CloudBees,Inc. All Rights Reserved. Automate the Builds Centralize and automate the entire build process ● Compiling the source code ● Packaging the compiled code ● Container images The build process should run for every commit.
  • 23.
    © 2019 CloudBees,Inc. All Rights Reserved. Automate the Builds The build should run as quickly as possible The builds should take no more than 5-10 minutes. Commit should be blocked during build, to support “stop-the-line” culture. If builds are long developers defer commits. ● Build ● Code scans, unit test ● Functional tests if possible Promote the build to extended CI/CD loop for longer activities
  • 24.
    © 2019 CloudBees,Inc. All Rights Reserved.© 2019 CloudBees, Inc. All Rights Reserved. Best Practice #6: Validate and Test
  • 25.
    © 2019 CloudBees,Inc. All Rights Reserved. Validate and Test Organizations that don’t validate every build are not practicing CI. Organizations that do not continuously test are not practicing CD. ● Manage tests like code or as code ● Fail if the build if test thresholds aren’t met ● Duplicate the production environment as much as possible
  • 26.
    © 2019 CloudBees,Inc. All Rights Reserved.© 2019 CloudBees, Inc. All Rights Reserved. Best Practice #10: Create a True DevOps Culture
  • 27.
    © 2019 CloudBees,Inc. All Rights Reserved. Create a True DevOps Culture This may be the most difficult practice to implement, but no DevOps transformation is possible without it. Starting a successful DevOps journey requires significant culture change, including: ● Executive support ● Training ● Funding ● A new mindset for everyone: the CIO/CTO, release managers, engineering managers, the ops team, and the dev team. But..start, learn, and improve
  • 28.
    © 2019 CloudBees,Inc. All Rights Reserved.© 2019 CloudBees, Inc. All Rights Reserved. Thank You Q & A

Editor's Notes

  • #4 This text is from the webinar registration page at https://webinars.devops.com/ci/cd-best-practices-for-your-devops-journey.
  • #6 I left a few slides from the SDM deck in here. I’m sure we don’t need all of them, but wasn’t sure how much you want to talk about SDM here.
  • #7 Liked the background on this slide and the next one, left it in here
  • #12 Brian
  • #13 Brian
  • #14 Doug
  • #15 Must Integrated to SCM, create a Bill of Materials, which is required Containers, pipeline as code, Confirg as code Allows you to observe
  • #16 Doug
  • #17 I did the rest of the slides in this format. It doesn’t take long to cut and paste the text into the bullet point style if you want to go that way. Brian: Do make note to the fact DVCS is Git, is the new way. But as you look to connect all teams you may have to build in support for a SVN style work flow Cover pull requests
  • #18 This is table stakes for continuous integration. A developer can set up an automated build and have the build run on every commit. But if the culture is to not commit frequently, it won’t matter. If a developer waits three weeks to commit or branches off for three weeks, he has delayed the integration and broken the principles. If a build breaks, the team has to sort through three weeks of work to figure out where it broke. Principle is integrate changes early and often to find and fix problems fast, so the cost of fixing issues is less, or even more importantly errors dont make it to production
  • #19 Do: Use solution like Multibranch plugin (Jenkins) or preview environment (Jenkins X) to trigger build on Pull Request
  • #24 NEED TO ADD TIPS ON HOW TO KEEP THE BUILDS SHORTER. But make sure they don’t contradict what we say in the Validate and Test section.