PuppetConf 2016: Enjoying the Journey from Puppet 3.x to 4.x – Rob Nelson, AT&T

Enjoying the Journey from
Puppet 3.x to Puppet 4.x
Rob Nelson

Who Am I?
Puppet user since 2014 (3.3 era)
Vox Pupuli, puppet-lint contributor
@rnelson0, https://rnelson0.com/

Agenda
•  Why upgrade?
•  Refactor our codebase for Puppet 4
•  Upgrade our Puppet master(s) and agents to Puppet 4.x
•  Refactor our codebase to remove Puppet 2- and 3-isms
•  Tips, tricks, and tools
•  Enjoying ourselves

Why?
•  Puppet 4 is old! First released March, 2015
•  Puppet 3 is really old! End Of Support on December 31, 2016
•  Puppet 4 only modules
•  Puppet 4 language improvements
•  Application Orchestration
•  PE first, FOSS eventually; some free implementations (such as choria) appearing
•  AIO Puppet and Puppetserver
•  Better performance, security; same agent/puppetserver between FOSS and PE
•  Puppet 5 is coming!

Who does this apply to?
•  Puppet Enterprise users
•  Puppet Opensource users
•  Foreman (1.13+) users
•  Master and Masterless

Blueprint
•  Start with Puppet 3.x
•  Read the release notes
•  Plan the roadmap
•  Validate / create tests
•  Refactor until your current version passes all tests
•  Upgrade/Replace the master(s)
•  Upgrade the agents
•  Repeat the Refactor / Upgrade steps until you get to 4.latest

Release Notes
•  All of them - not just the latest version
•  Identify potential issues, deprecated features, etc
•  Determine the minimum version required to upgrade to target version
•  Stay up to date

Define the Roadmap
•  Determine the current version
•  In-place upgrades or new infrastructure?
•  Identify intermediate version steps
•  Enable Future Parser [and Strict Variables] before you hit 4.x
•  PE: Requires intermediate upgrades or fresh installs (check KB)
•  FOSS: Go straight to 4.latest
•  Determine how upgrades and interruptions affect ecosystem products – PE
Console/puppetboard, SEIMs, monitoring, etc.

FOSS Example Roadmap
•  3.6.0 -> 3.8.7
•  3.8.7 w/Future Parser [and Strict Variables]
•  3.8.7 -> 4.7.0
•  Today’s example roadmap

PE Example Roadmap
•  3.7.2 -> 3.8.6
•  3.8.6 w/Future Parser [and Strict Variables]
•  3.8.6 -> 2015.3.3
•  2015.3.3 -> 2016.2.1

Validate/Create Tests
•  Tests assure (mostly) predictable behavior
•  Determine what kinds of tests you need - unit, acceptance, integration, other?
•  Good testing setup in puppet-module-skeleton
•  Never written an rspec-puppet test? puppet-retrospec generates naive tests
that need tuned
•  Existing tests must pass before modifying code
•  Turn on Future Parser [and Strict Variables] only at 3.8.x
•  Beyond tests: catalog diffs, personalized tests

Rspec Tests
$ cat spec/classes/apache_spec.rb
require 'spec_helper'
describe 'profile::apache', :type => :class do
let :facts do
{
facts_hash
}
end
context 'with defaults for all parameters' do
it { is_expected.to create_class('profile::apache') }
it { is_expected.to contain_package('httpd') }
it { is_expected.to contain_user("apache") }
end
end

Rspec Run
[rnelson0@build03 profile:production]$ bundle exec rspec spec/classes/
apache_spec.rb
profile::apache
with defaults for all parameters
should contain Class[profile::apache]
should contain Package[httpd]
should contain User[apache]
Finished in 7.82 seconds (files took 2.49 seconds to load)
3 examples, 0 failures

Refactor
•  Create a new branch for the target version, e.g. 3.8.7
•  Test against current and target versions, e.g. ~>3.6.0 and ~>3.8.0, with and
without Future Parser/Strict Variables
•  Identify failing tests, refactor to fix
•  Upgrade modules as early as possible. Be aware of the required Puppet
version for a module version, and look out for defunct or migrated modules,
such as those transferred to Vox Pupuli
•  Move forward when tests are green for current AND next version

Testing with particular Puppet versions
$ grep PUPPET Gemfile
gem "puppet", ENV['PUPPET_GEM_VERSION'] || '~> 4.0'
[rnelson0@build controlrepo]$ export PUPPET_GEM_VERSION='~>3.8.0'
[rnelson0@build controlrepo]$ bundle update
Installing puppet 3.8.7 (was 4.6.0)
[rnelson0@build controlrepo]$ bundle exec puppet --version
3.8.7
[rnelson0@build controlrepo]$ export PUPPET_GEM_VERSION='3.8.1'
[rnelson0@build controlrepo]$ bundle update
Installing puppet 3.8.1 (was 3.8.7)
[rnelson0@build controlrepo]$ bundle exec puppet --version
3.8.1

High level Master(s) upgrade process
•  Prep for new master/in-place upgrade
•  Deploy new/upgrade in testing
•  Revert
•  Deploy new/upgrade in production
•  Start with Master of Masters or other “parent” nodes first
•  Update separate PuppetDB node, puppetdb-termini on masters

Replace the Master
•  Prepare a new operational environment
•  Do not serve bad/incorrect catalogs to existing nodes
•  Deploy a new master on the target puppet version
•  Bootstrap configuration/code
•  Test the master against itself, puppet agent -t
•  Deploy and test canary nodes in the same operational environment

In-place Master upgrade
•  Snapshot (or equivalent) the master(s) and canary nodes
•  Restrict access to the master:
•  Control access with firewall/load balancer
•  Disable puppet agent on nodes with orchestration
•  Revoke certificates for non-canary nodes
•  Revoke the CA, generate a new CA and new agent certs for canary nodes only
•  Upgrade the master
•  Test the master then canary nodes with puppet agent -t

Troubleshooting
•  Collect logs from the master and canaries
•  Look for changed resources, number of resources in catalog
•  Revert production environment
•  Analyze cause(s)
•  Refactor code and data to address issues
•  Try again
•  Learn from failures, prevent them in the future

Upgrade the Agents
•  Can often skip on PATCH versions and some MINOR versions (see rel notes)
•  puppetlabs/puppet_agent (requirements) updates agents on next check-in
•  Orchestration
•  Replace nodes with new instances running the new agent
•  By hand

Repeat
•  Relax, enjoy the success of an upgrade!
•  Start working on the next version/feature flags
•  Repeat the Refactor / Upgrade steps

Keeping Up
Refactor to take advantage of Puppet 4 language improvements, new tools (ex:
r10k -> PE Code Manager), new file locations, etc.
•  PE has quarterly upgrades, FOSS more frequent
•  The less frequently you do something, the more painful it is. “Upgrade early
and upgrade often!”
•  Try not to get more than 2 MINORs behind
•  Test against puppet version ~>4.0 (latest v4) and run bundle update before
manual tests

Puppet 4 Language Improvements
•  Replace create_resources() with iteration
•  Replace validate_*() with data types (including a Sensitive type)
•  There is a validate_legacy() helper function available in puppetlabs/stdlib to
assist with replacing validate_*() functions (blog)
•  Simplified resource wrappers with * and + operators
•  Improved default attributes are per-expression
•  New template type EPP is available
•  Puppet Lookup, Data In Modules, and other hiera improvements
•  Use $facts[] instead of global variables to tidy up the namespace and remove
ambiguity

Tips & Tricks – Puppet Enterprise
•  PE includes support, use it for planning/errors
•  Puppet Enterprise Upgrade Service to engage Pro Services
•  PE Classifier changes over time. Review
Preconfigured Node Groups documentation
•  pe_puppetserver_gem is out, puppetserver_gem is in
•  Do not use PE’s bundled Ruby for other Ruby tasks, conflicts between
bundled/downloaded gems. Recommend rbenv/rvm or SCL-equiv instead
•  Do not ever do this on your master. EVER!

Tips & Tricks - Strings
Understand how string conversion works in puppet, hiera, rspec-puppet, and how it
has changed:
•  rspec-puppet: 'undef' represents an undefined value
•  Puppet DSL: it is the string undef! Try :undef, without quotes, instead
•  If you have a file resource with a title or path of ${undefvar}/${populatedvar},
rspec will start failing because file { 'undef/etc/app.conf' :} is not valid
•  Similar issue with 'true' vs true and 'false' vs false
•  Other common issues: input from hiera/ENC, quoted numbers as strings, stringify
vs structured facts, unquoted strings in case selectors, etc
•  May require acceptance tests/canary nodes to become apparent

Tips & Tricks - Hiera
•  Hiera eyaml gem is lost during the upgrade to the 4.x puppetserver
•  Enable the yaml backend and ensure that the master does not rely on eyaml data
•  Run the agent on the master to redeploy the gem (with puppet/hiera or similar)
before agents check in
•  %{}: used to prevent variable interpolation, as in %%{}{environment} to generate
the string %{environment}. In 3.x and in 4.5 resolves to an empty string, in 4.0-4.4
it returned the scope, giving strings like %<#Hiera:7329A802#>{environment}. Use
%{::} instead, as in %%{::}{environment}. Affects PE < 2016.2.0
•  datadir: some versions expect :: prepends to variables and others do not.
Change %{environment} to %{::environment}. Likely PE < 2016.2.0 as well

Tips & Tricks - Other
•  Review modules and their supported versions. May be incorrect or weak
assumptions (>= 3 but should also include < 4 – check tests)
•  Upgrades across major versions mean additional troubleshooting
•  Upgrade early – but with caveats
•  Many tools to assist with automating version upgrades in your Puppetfile
•  ERB scope: prepend most variables with @ (<%= var %> to <%= @var %>)
•  Script to detect usage of hardcoded /etc/puppet paths, no longer correct in v4
•  External fact weighting bug: FACT-1413
•  Minimize coupled/entangled changes
•  Ask for help! Colleagues, social media, etc.

Tools
•  Puppet Community Slack / IRC and Mailing Lists
•  puppet-ghostbuster helps you find "dead code" that you may want to prune
before you start on your refactoring journey.
•  rspec-puppet, puppetlabs_spec_helper, and puppet-lint are improving their
Puppet 4 support
•  A number of catalog diff tools exist (diff generators and a viewer) to inspect
the actual catalog differences from active nodes across different versions of
Puppet.

Links
Additional information on Puppet 4 and Migrations
•  Official Puppet Upgrade Docs
•  Whirlwind Tour of Puppet 4 by R.I. Pienaar
•  The Power of Puppet 4 by Martin Alfke
•  Puppet - our journey from Puppet 3.8 to Puppet 4 by Jonas Genannt

Summary
•  Plan the upgrade blueprint
•  Have working tests before upgrading
•  Step through the new versions / feature flags
•  Refactor code to take advantage of the language and tool improvements
•  Keep mowing
•  Enjoy the journey!

PuppetConf 2016: Enjoying the Journey from Puppet 3.x to 4.x – Rob Nelson, AT&T

More Related Content

What's hot (18)

Viewers also liked (12)

Similar to PuppetConf 2016: Enjoying the Journey from Puppet 3.x to 4.x – Rob Nelson, AT&T (20)

More from Puppet (20)

Recently uploaded (20)

PuppetConf 2016: Enjoying the Journey from Puppet 3.x to 4.x – Rob Nelson, AT&T