Performance Tuning Your Puppet Infrastructure - PuppetConf 2014
5 likes•3,037 views
The document discusses ways to monitor and tune Puppet infrastructure using the same techniques used for applications. It describes instrumenting the Puppet master and database with New Relic to monitor performance. It also discusses collecting logs and reports from Puppet agents and masters and sending them to Elasticsearch for analysis in Kibana.
![Getting more detail
• The Ruby Agent’s MethodTracer
allows us to get details of what is
going on inside a Transaction.
• By hooking into Puppet’s built-in
Profiler calls, we can quickly
instrument all the most important
things.
• This example is for Puppet 3.3.1. In
newer versions it is much easier.
require
"puppet/util/profiler"
module
Puppet::Util::Profiler
include
::NewRelic::Agent::MethodTracer
self.profile(message,
&block)
if
/^Processed
request
def
/
===
message
#
Top
level
trace,
skip
for
now
yield
else
metric_name
=
self.message_to_metric(message)
trace_execution_scoped([metric_name])
do
yield
end
end
end
def
self.message_to_metric(message)
message.chomp!
message.gsub!(/^(Compiled
catalog)
for
.*/,
'1')
message.gsub!(/^(Filtered
result
for
catalog)
.*/,
'1')
message.gsub!(/(d+)
/,
'')
message.gsub!(/^(Evaluated
resource)
/,
'1/')
message.gsub!(/[.*]/,'')
message.gsub!(/:s/,'/')
message.gsub!(/^(Called)
/,
'1/')
"Custom/Puppet/#{message}"
end
end](https://image.slidesharecdn.com/performancetuningyourpuppetinfrastructure-nicbendersnewrelic-140925182021-phpapp02/85/Performance-Tuning-Your-Puppet-Infrastructure-PuppetConf-2014-15-320.jpg)
Performance Tuning Your Puppet Infrastructure - PuppetConf 2014
- 1. Performance Tuning Your (Puppet) Infrastructure Nic Benders OWNING http://nicbenders.com/presentations/puppetconf-2014
- 2. I work at New Relic. ! This is a story about treating your infrastructure the same way you treat your applications. ! This is also a story about how we use New Relic, at New Relic. http://nicbenders.com/presentations/puppetconf-2014
- 3. Managing Our Applications • Source Control • Continuous Deployment • Performance Monitoring • Operational Analytics • Log Collection GitHub with Pull Requests Jenkins + Capistrano New Relic APM New Relic Insights Heka + ElasticSearch + Kibana
- 4. Managing Our Infrastructure • Source Control • Continuous Deployment • Performance Monitoring • Operational Analytics • Log Collection GitHub with Pull Requests Jenkins + Capistrano New Relic APM New Relic Insights Heka + ElasticSearch + Kibana
- 10. DON’T PANIC • These example are for Puppet Enterprise 3.1, but this will work with Puppet Open Source, or with newer versions. You’ll just need to make a few changes here and there. • If you don’t have a New Relic account, you can sign-up for a free “Lite” account. • All of the examples are up in a more useful format at the URL below. (The URL will also be shown at the end of the presentation.) http://nicbenders.com/presentations/puppetconf-2014
- 11. Instrumenting the Puppet Master • Install newrelic_rpm using the Puppet’s Ruby interpreter • Modify the config.ru in the Puppet Master root dir • Create a newrelic.yml in the same directory # /var/opt/lib/pe-‐puppetmaster/config.ru ! # Setup the New Relic Ruby Agent before anything else require 'rubygems' require 'newrelic_rpm' # END New Relic Ruby Agent setup # /var/opt/lib/pe-‐puppetmaster/newrelic.yml ! common: &default_settings license_key: '000-‐your-‐license-‐key-‐here-‐000' app_name: "Puppet Master" log_level: info log_file_path: '/var/log/pe-‐puppet/' ssl: false capture_params: true transaction_tracer: enabled: true error_collector: enabled: true ignore_errors: "ActionController::RoutingError" production: <<: *default_settings monitor_mode: true
- 13. Getting good transaction names • Use set_transaction_name in a Rack Middleware to tell the Ruby Agent how we are going to name the “web transactions” for the Puppet Master • Use add_custom_parameters to pass the environment and other useful data along # # We don't want every transaction to just say "call", so define a # Transaction "namer" for the Ruby Agent based on the URL structure # class CoolNamer def initialize(app, opts = {}) @app = app end def call(env) req = Rack::Request.new(env) _, environment, kind_of_thing, resource = req.path.split('/',4) NewRelic::Agent.set_transaction_name(kind_of_thing, :category => :rack) NewRelic::Agent.add_custom_parameters(:puppet_env => environment) NewRelic::Agent.add_custom_parameters(:puppet_resource => resource) @app.call(env) end end use CoolNamer # END Transaction "namer"
- 15. Getting more detail • The Ruby Agent’s MethodTracer allows us to get details of what is going on inside a Transaction. • By hooking into Puppet’s built-in Profiler calls, we can quickly instrument all the most important things. • This example is for Puppet 3.3.1. In newer versions it is much easier. require "puppet/util/profiler" module Puppet::Util::Profiler include ::NewRelic::Agent::MethodTracer self.profile(message, &block) if /^Processed request def / === message # Top level trace, skip for now yield else metric_name = self.message_to_metric(message) trace_execution_scoped([metric_name]) do yield end end end def self.message_to_metric(message) message.chomp! message.gsub!(/^(Compiled catalog) for .*/, '1') message.gsub!(/^(Filtered result for catalog) .*/, '1') message.gsub!(/(d+) /, '') message.gsub!(/^(Evaluated resource) /, '1/') message.gsub!(/[.*]/,'') message.gsub!(/:s/,'/') message.gsub!(/^(Called) /, '1/') "Custom/Puppet/#{message}" end end
- 17. Instrumenting the Puppet DB • Download the Java Agent from your “Account Settings” page. • Create an /opt/puppet/newrelic dir and place the newrelic.jar and newrelic.yaml file into it. • Modify the newrelic.yaml • Modify JAVA_ARGS # /etc/syslog/pe-‐puppetdb ! # Enable the New Relic Java Agent by adding this # to the bottom of the file JAVA_ARGS="-‐javaagent:/opt/puppet/newrelic/newrelic.jar ${JAVA_ARGS}" # /opt/puppet/newrelic/newrelic.yml ! common: &default_settings license_key: '000-‐your-‐license-‐key-‐here-‐000' ! app_name: 'Puppet DB' log_level: info log_file_path: /var/log/pe-‐puppetdb/ ssl: false transaction_tracer: enabled: true transaction_threshold: apdex_f record_sql: obfuscated stack_trace_threshold: 0.2 explain_enabled: true explain_threshold: 0.2 browser_monitoring: auto_instrument: false production: <<: *default_settings
- 22. Log Collection
- 23. Puppet Agent already goes into Syslog • Agent syslogs have a ton of useful information. We just need to get it to a useful place. • Our application and syslog data is collected by a Heka process on each server. It then makes its way to ElasticSearch, where it can be queried by Kibana.
- 24. Send run reports to the same place • Add your log system as a Report processor! • We use https://github.com/ logstash/puppet-logstash-reporter • To get the data into Heka, we had to write some Lua, but it was pretty straightforward.
- 25. pe_puppet::master::config_hash: reports: 'http,puppetdb,logstash' ! logstash_reporter::logstash_host: 'our-‐heka-‐server' # Logstash-‐style report processer through Heka class { 'logstash_reporter': logstash_port => '22253', logstash_host => hiera('logstash_reporter::logstash_host'), }
- 26. Managing Our Infrastructure • Source Control • Continuous Deployment • Performance Monitoring • Operational Analytics • Log Collection GitHub with Pull Requests Jenkins + Capistrano New Relic APM New Relic Insights Heka + ElasticSearch + Kibana
- 27. We’re Hiring Site Reliability Engineers MySQL Experts Engineering Managers Data Pipeline Engineers newrelic.com/about/careers