Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

W11

DevOps
&
Testing

5/4/16
13:45

Continuous
Integration
Testing

Techniques
to
Improve
Chef
Cookbook

Quality

Presented
by:

Glenn
Buckholz

Coveros,
Inc.

Brought
to
you
by:

350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073

888-268-8770 904-278-0524 info@techwell.com http://www.stareast.techwell.com/

Glenn
Buckholz

Coveros,
Inc.

With
fifteen
years
of
industry
experience,
Glenn
Buckholz
leads
continuous

integration
and
deployment
automation
efforts
at
Coveros.
His
career
began
as
a

consultant
implementing
automated
test
frameworks
and
introducing
the

concept
of
change
management
to
many,
many
projects.
Glenn
then
decided
to

become
a
part
of
honest
society
and
settled
down
at
the
Public
Company

Accounting
Oversight
Board
as
their
full8 time
enterprise
change
manager.

Several
years
later,
he
joined
Coveros,
where
he
specializes
in
implementing
agile

practices
and
CI,
and
engineering
configuration
management
instead
of
simply

documenting
it.

© Copyright 2016 Coveros, Inc. All rights reserved.
Continuous Integration Testing
Techniques to Improve Chef
Cookbook Quality
STAREAST – 04 May 2016
Glenn Buckholz
glenn.buckholz@coveros.com

Agenda
•What did I do?
•What is problem?
•What are the needs?
•Demo
•Architecture
•Workflow
•Testing Techniques and Tools
•Metrics?
•Docker and Cloud
•Discussion

What did I do?
● Used a modern CI/CD toolchain to apply
continuous integration testing techniques to
Chef code (CI):
○ Static analysis
○ Chefspec testing
○ Automated Functional Testing
● Introduces the idea of automatically making
cookbooks available when they are ready
(CD).
○ There is still a human gate between
available and in use by way of version
pinning.

What is the problem?
● Infrastructure as Code is currently the wild
west with little formal testing methodology.
● Chef lacks coherent CM on its own.
○ Cookbook versions and freezing are not
enough.
● No quick way to advance a piece of
infrastructure from one state to another and
back for purposes of testing.
● No minimal standards and requirements on
cookbook functionality.

What are the needs?
YOU ARE CHANGING THE TIRE AT
55MPH AND CAN’T STOP!!!
● Find flaws early.
● Ability to test pieces of an architecture
without having to have a whole system.
○ Unit testing system components.
● Simulate the effect of a deploy on a running
system.
● Dashboarding so you can trend code
quality.

Definitions
● Hands Off Deploy - Deployment that is
completely driven by Chef or some other
infrastructure tool.
● Static Analysis - evaluation of the code
without execution.
● Feedback loop - The events between when
a developer commits their code and results
can be observed.
● Application Code - The custom or home
grown software the current IT organization
is curating.

Demo

Architecture
Local Git Branch
Chefspec
Test Kitchen
Food Critic
Developer Workstation
Docker
Container
Binary Repository
Docker
Repository
Local Testing
Docker
Container
Upload Cookbook

Workflow
1. Clone master branch on the developer
machine.
2. Make changes and run static analysis.
3. Run Chefspec and test kitchen locally on a
docker container.
4. Push changes to remote branch.
5. Jenkins detects changes and checks out
branch onto a slave node.

Workflow
6. Jenkins merges the code with master
locally.
7. Chef reruns the static analysis.
8. Chef reruns Chefspec and test kitchen.
9. Results and metrics are recorded.
10.Successful tests see code merged into
master.
11.Cookbooks are uploaded to the chef
server.
12.After the code is executed in production,
AMI and Docker containers are
automatically updated from production.
Steps 8-9
automate
integration
testing

Reuse of well known patterns
● Feature Branches
○ Small bite sized changes.
● Automated Merging
○ Let the Jenkins objectively merge the
code after certain minimum standards are
achieved by the developer.
● Local tests that mirror a golden standard
○ Give the developers a cheat sheet.
● Short feedback loops.
● Auto Promotion
○ Let Jenkins keep the working cookbooks
in sync with SCM
Smaller changes are
more likely to make it
through the pipeline.

Testing Techniques
● Tabula Rasa
○ Start with a base image
○ Apply Security Hardening
○ Install stack fresh
○ Deploy application code
○ Run tests
● Production Approximation
○ Start with a clone from production
○ Run cookbooks
■ Only changes are executed
○ Run test

Testing Techniques
● Test in parallel
○ Do not block on critical resources clone
them.
○ VMs and containers are cheap, create
one for every commit you want to test.
○ Make unit and integration test cases
isolated. (Script system state for each
test case do not rely on other tests)
● Test local
○ Very short feedback loop for developers.
○ Reduces most pipeline failures to
integration errors.

Testing Tools
● Unit Testing
○ chefspec - extension of Rspec
● Functional Testing
○ BATS or chefspec
● Static Analysis
○ Foodcritic - lint like
● Git hooks
○ Make certain static tests run as a git hook
making them prerequisites to commit or
push.

Testing Tools
● Git Branching
○ Let the branch name decide if the CI
stack will run the changes.
○ You may want to use a prefix CI-
branchname

Testing Tools - Jenkins Master
● Jenkins master only monitors git.
○ Container on slave in the cloud for quick
tests.
○ VMs in cloud for executing tests in a
production like system.
■ Template AMI matches target machine.
○ All “work” takes place off the master to allow
testing in parallel.
● Create Jenkins jobs to capture production
templates post deployment. (AMI or docker
image)

Testing Tools - Developer Workstation
● Provide an easy to install package that includes:
○ Tools - Chefspec, Foodcritic, etc…
○ Docker settings.
○ Access to a docker repository.
○ Templates to create new cookbooks
● Enough memory and CPU to run the tools.
● DO NOT ALLOW COOKBOOK UPLOAD PRIVILEGES!!!!!
● DO NOT ALLOW MERGE TO MASTER PRIVILEGES!!!!!
● The ability for a developer to find mistakes locally is the
most cost effective testing technique available to an
organization. (Short Feedback Loop)

Transitioning to the CI Approach
● From the previous slides we have the tools to create
an method for retrofitting a CI driven approach to an
existing Chef infrastructure.
● On a separate branch create cookbooks that install
a piece of the production stack on a server or
container.
● On a feature branch develop just the chef code
needed to deploy your custom application.
● Slowly merge the two once per iteration. Merge one
item from the stack branch and merge your install
code to master.

Transitioning to the CI Approach
● Do not try to boil the ocean.
● Start with installing the application code.
● Move up to controlling the application stack
one layer at a time.
● This maps to Agile sprints and iterations.
● Cookbooks should be idempotent.
Master (What is in Chef Server)
Install Code
(Feature
Branch)
Install Code
(Feature
Branch)
Install Code
(Feature
Branch)
Install Code
(Feature
Branch)
Install Code
(Feature
Branch)
Infrastructure or Application Stack Cookbooks (Apache, Tomcat, etc.)
Test
Here
Test
Here
Test
Here
Test
Here

Metrics
● Failed Merges
○ How many times did a developer cause a
potential merge conflict?
● Static analysis thresholds
○ How many times did a commit fail?
● Failed rspec tests.
● Failed functional tests.
● How long is the component down during
upgrade.

Docker and Cloud
● Docker
○ Can be very fast
○ Need a local Docker repository.
● Cloud
○ You can use a mirror of production
environment.
○ Need to find a way to clone VMs without
downtime or cluster components.
● If production uses docker this is a moot
point, just use docker.

Discussion Points
● Tool chain can be complicated.
○ If you already invested in CI you have a sunk cost
with many of the tools already.
● Two major types of transitions:
○ Introducing CI to chef.
○ Introducing Chef and CI to a project.
● Start small, big bang automation is high risk.
● Make the quality gates automated and give the
developers the answer key, don’t change the test, and
let Jenkins proctor the exam.
● Allocate time to sync production changes to testing
images.

What Problems Did We Solve?
● Version management - The git cookbook version linkage
provides change traceability. This is enforced by Jenkins.
● Enforcing testing in general for Infrastructure - By ceeding
cookbook upload control to Jenkins we can have a
minimal set of quality standards for all deployments and
changes.
● Making Testing accessible to the developers - By putting a
framework at their fingertips. This makes it easier for
developers to follow the process than circumvent the
process.
● Provide testing metrics so cookbook quality can be
analyzed.
● Provided an automated testing framework to provide quick
feedback on integration errors.
● Disposable infrastructure for rapidly changing server state.

What Problems Didn’t We Solve?
● Adoption
○ Change it hard, things can be done to ease
the transition but it cannot be forced. You
still have to convince developers the value
proposition of writing the tests.
● Integration errors
○ While we can find them faster, there is no
tool I know of that can predict code
interaction accurately.

Thoughts? Questions?
Thank you for your time.

Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

More Related Content

What's hot (20)

Similar to Continuous Integration Testing Techniques to Improve Chef Cookbook Quality (20)

More from Josiah Renaudin (20)

Recently uploaded (20)

Continuous Integration Testing Techniques to Improve Chef Cookbook Quality