developing sysadmin, sysadmining developersGuug devops puppet
0 likes967 views
This document summarizes Martin Alfke's presentation on developing sysadmin skills and sysadmining developers using Puppet. The presentation covered topics like environments, modules, templates, Augeas, multi-master Puppet configurations, running Puppet without a master, using ENC and Hiera for node and data classification, PuppetDB for storing configurations and exported resources, and dashboards for infrastructure management and monitoring.
![Environments
• puppet.conf
[master]
[test]
manifest = /etc/puppet/test/manifests/site.pp
modulepath = /etc/puppet/test/modules
[mailteam]
manifest = /etc/puppet/mail/manifests/site.pp
modulepath = /etc/puppet/mail/modules
[agent]
environment = test
Environments - Modules - Templates - Augeas - Master - Masterless
© Martin Alfke - 2012
Freitag, 2. November 12](https://image.slidesharecdn.com/guugdevopspuppet-121102112749-phpapp02/85/developing-sysadmin-sysadmining-developersGuug-devops-puppet-5-320.jpg)
![Environments
• Each environment may have multiple modulepaths
[master]
[test]
manifest = /etc/puppet/test/manifests/site.pp
modulepath = /etc/puppet/test/modules:/data/puppet/team/test/modules
[mailteam]
manifest = /etc/puppet/mail/manifests/site.pp
modulepath = /etc/puppet/mail/modules:/data/puppet/team/core/modules
Environments - Modules - Templates - Augeas - Master - Masterless
© Martin Alfke - 2012
Freitag, 2. November 12](https://image.slidesharecdn.com/guugdevopspuppet-121102112749-phpapp02/85/developing-sysadmin-sysadmining-developersGuug-devops-puppet-6-320.jpg)
![Multi Master
• separate puppet ca and puppet master
puppet.conf on puppet ca (single instance)
[master]
ca = true
puppet.conf on puppet master
[master]
ca = false
puppet.conf on agent
[agent]
ca_server = <puppet ca>
server = <puppet master>
Environments - Modules - Templates - Augeas - Master - Masterless
© Martin Alfke - 2012
Freitag, 2. November 12](https://image.slidesharecdn.com/guugdevopspuppet-121102112749-phpapp02/85/developing-sysadmin-sysadmining-developersGuug-devops-puppet-24-320.jpg)
![ENC and Hiera
• ENC in puppet
/etc/puppet/puppet.conf
[master]
node_terminus = exec
external_nodes = /etc/puppet/bin/my_great_enc.exe
ENC and Hiera - Puppet DB - Dashboard
© Martin Alfke - 2012
Freitag, 2. November 12](https://image.slidesharecdn.com/guugdevopspuppet-121102112749-phpapp02/85/developing-sysadmin-sysadmining-developersGuug-devops-puppet-36-320.jpg)
developing sysadmin, sysadmining developersGuug devops puppet
- 1. developing sysadmin - sysadmining developers develop your platform and your application management GUUG Berlin 01.11.2012 Martin Alfke <[email protected]> © Martin Alfke - 2012 Freitag, 2. November 12
- 2. Agenda puppet environments puppet modules puppet templates puppet and augeas puppet multi master puppet without master © Martin Alfke - 2012 Freitag, 2. November 12
- 3. Environments “admin’s and dev’s cooperate!” Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 4. Environments • Split up modules into several repositories • “production” is default and always there • Naming is abritrary • Master needs to know about environments • Client needs to send environment information Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 5. Environments • puppet.conf [master] [test] manifest = /etc/puppet/test/manifests/site.pp modulepath = /etc/puppet/test/modules [mailteam] manifest = /etc/puppet/mail/manifests/site.pp modulepath = /etc/puppet/mail/modules [agent] environment = test Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 6. Environments • Each environment may have multiple modulepaths [master] [test] manifest = /etc/puppet/test/manifests/site.pp modulepath = /etc/puppet/test/modules:/data/puppet/team/test/modules [mailteam] manifest = /etc/puppet/mail/manifests/site.pp modulepath = /etc/puppet/mail/modules:/data/puppet/team/core/modules Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 7. Modules “plug things together simple” Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 8. Modules • Difference between modules and classes • Module: • strict directory naming for autoloading • each module has at least one class • Class: • available but not applied automatically Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 9. Modules • directory structure /etc/puppet/test/modules/ <-- modulepath apache/ <-- modulename manifests/ <-- manifests path within module init.pp <-- initial class fetched from autoloader server.pp <-- additional class(es) files/ <-- directory for module file serving templates/ <-- directory for module templates lib/ <-- directory for facts or functions tests/ <-- directory for tests during develop Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 10. Modules • class, file and template naming structure /etc/puppet/test/modules/ apache/ manifests/ init.pp <-- class apache { ... } server.pp <-- class apache::server { ... } files/ <-- “puppet:///modules/apache/<filename>” templates/ <-- template(‘apache/<filename>’) lib/ tests/ Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 11. Modules • class structure class apache { package { ‘apache2’: ensure => present, } file { ‘/etc/apache2/apache2.conf’: content => template(‘apache/apache2.conf.erb’), } file { ‘/etc/apache2/conf.d/charset’: source => ‘puppet:///modules/apache/charset’, } service { ‘apache2’: ensure => running, } } Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 12. Modules • use classes node ‘www01.domain.tld’ { class { ‘apache’: } <-- old: include apache } Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 13. Modules • resources, classes, parameterized classes resource_type { ‘title’: attribute => value, } class <title> { ... } class <title> ( $variable = value) { ... } class { ‘<title>’: } class { ‘<title>’: variable => value, } Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 14. Modules • using ruby in classes /etc/puppet/test/modules/apache/manifests/init.rb hostclass :apache do package :apache2, :ensure => present package :libapache2-php, :ensure => present service :apache2, :ensure => running end Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 15. Templates “code your config” Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 16. Templates • Ruby ERB template engine • Normally requires in-depth configuration review • Be aware of variable scoping ! Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 17. Templates • use variables from puppet in templates $ntpserver = ‘10.2.3.4’ file { ‘/etc/ntp.conf’: content => template(‘ntp/ntp.conf.erb’), } # ntp.conf.erb <% if @ntpserver %> <-- old: if has_variable(‘ntpserver’) server <%= @ntpserver %> <-- @ syntax is new. uses current scope <% else %> server pool.ntp.org <% end %> Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 18. Augeas “clean your lenses” Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 19. Augeas • Make changes to single lines • Do not manage the complete configuration file in puppet Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 20. Augeas • augeas uses lenses to split up config files augtool print /files/etc/sysctl.conf augtool set net.ipv4.forward 1 augeas { ‘set_ipv4_forward’: context => ‘/files/etc/sysctl.conf’, changes => “set net.ipv4.forward 1”, } Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 21. Augeas • Attention! • Not all configuration files are supported ! • Augeas needs key-value pairs • Within puppet ruby-augeas extension is required Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 22. Multi Master “no one can serve two masters!” Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 23. Multi Master • Load-Balancing with SSL separation • several Data Center • do you really have more than 1000 nodes? Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 24. Multi Master • separate puppet ca and puppet master puppet.conf on puppet ca (single instance) [master] ca = true puppet.conf on puppet master [master] ca = false puppet.conf on agent [agent] ca_server = <puppet ca> server = <puppet master> Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 25. Multi Master • use multiple master (without ca) • apache/nginx and loadbalancing • ipvsadm Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 26. Multi Master • use multiple master (without ca) • pros: • file serving handled better • more masters compile catalogs • cons: • single ca only Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 27. Multi Master • avoid multiple masters • use templates ! • templates are generated on the master during catalog compilation • files needs to get fetched by the nodes • use mod_passenger Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 28. without Master “you shall have no master” Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 29. without Master • Pre-compile catalogs • Run puppet apply locally Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 30. without Master • Very large environment (>15.000 nodes) • Multiple locations world wide Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 31. without Master • Compile catalogs for all nodes on master • puppet master compile <fqdn> • Copy catalogs to nodes to execute them • puppet apply --catalog <catalog file name> Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 32. without Master • Pros: • no dedicated master, catalog compilation may take place everywhere • Cons: • no modules !! • fileserving has to be done locally (e.g. NFS mount) Environments - Modules - Templates - Augeas - Master - Masterless © Martin Alfke - 2012 Freitag, 2. November 12
- 33. developing sysadmin - sysadmining developers Wait ! There is more ! Martin Alfke <[email protected]> © Martin Alfke - 2012 Freitag, 2. November 12
- 34. ENC and Hiera “data, data, data” ENC and Hiera - Puppet DB - Dashboard © Martin Alfke - 2012 Freitag, 2. November 12
- 35. ENC and Hiera • External Nodes Classifier • executable with one parameter (fqdn) • can be anything • get info from filesystem: cat $1.yaml • get info from inventory database ENC and Hiera - Puppet DB - Dashboard © Martin Alfke - 2012 Freitag, 2. November 12
- 36. ENC and Hiera • ENC in puppet /etc/puppet/puppet.conf [master] node_terminus = exec external_nodes = /etc/puppet/bin/my_great_enc.exe ENC and Hiera - Puppet DB - Dashboard © Martin Alfke - 2012 Freitag, 2. November 12
- 37. ENC and Hiera • /etc/puppet/bin/my_great_enc.exe www1.domain.tld --- parameters: location: de-ber2 classes: ntp: ntpserver: 10.2.2.2 apache: mysql: environment: production ENC and Hiera - Puppet DB - Dashboard © Martin Alfke - 2012 Freitag, 2. November 12
- 38. ENC and Hiera • Hiera - hierarchial data structure • built in into Puppet 3.x • add-on in Puppet 2.7.x ENC and Hiera - Puppet DB - Dashboard © Martin Alfke - 2012 Freitag, 2. November 12
- 39. ENC and Hiera • Hiera configuration /etc/puppet/hiera.yaml :hierarchy: - %{operatingsystem} - common - %{datacenter} - %{serverfunction} :backends: - yaml :yaml: :datadir: ‘/etc/puppet/hieradata’ ENC and Hiera - Puppet DB - Dashboard © Martin Alfke - 2012 Freitag, 2. November 12
- 40. ENC and Hiera • Hiera data /etc/puppet/hieradata/debian.yaml --- ssh_packages: - ‘openssh-server’ - ‘openssh-client’ - ‘openssh-blacklist’ /etc/puppet/hieradata/centos.yaml --- ssh_packages: - ‘openssh’ - ‘openssh-clients’ - ‘openssh-server’ ENC and Hiera - Puppet DB - Dashboard © Martin Alfke - 2012 Freitag, 2. November 12
- 41. ENC and Hiera • Hiera usage /etc/puppet/modules/ssh/manifests/init.pp class ssh { $ssh_packages = hiera(‘ssh_packages’) package { “${ssh_packages}”: ensure => present } } ENC and Hiera - Puppet DB - Dashboard © Martin Alfke - 2012 Freitag, 2. November 12
- 42. PuppetDB “getting it all” ENC and Hiera - Puppet DB - Dashboard © Martin Alfke - 2012 Freitag, 2. November 12
- 43. PuppetDB • Used for storeconfigs and exported resources • Schema for PostgreSQL • Will get more features soon ENC and Hiera - Puppet DB - Dashboard © Martin Alfke - 2012 Freitag, 2. November 12
- 44. Dashboard “management needs graphs” ENC and Hiera - Puppet DB - Dashboard © Martin Alfke - 2012 Freitag, 2. November 12
- 45. Dashboard • Open Source Dashboard is dead end • PuppetLabs is working on two other tools ENC and Hiera - Puppet DB - Dashboard © Martin Alfke - 2012 Freitag, 2. November 12
- 46. developing sysadmin - sysadmining developers Questions? Martin Alfke <[email protected]> © Martin Alfke - 2012 Freitag, 2. November 12