Automating complex infrastructures with Puppet

Puppetizing Complex
Applications
with sipXecs as an example

Kris Buytaert

Kris Buytaert
● I used to be a Dev, Then Became an Op
● Senior Linux and Open Source Consultant
@inuits.be
● „Infrastructure Architect“
● Building Clouds since before the Cloud
● Surviving the 10th floor test
● Co-Author of some books
● Guest Editor at some sites

Today

● About SIPX
● About Puppet
● Deploying SipX
● ...

Not quite a Muppet...

● Puppet is...
● OSS
● A DSL language
● Written in Ruby
● Client/server oriented
● Contains abstraction layers
● Repeatable processes

Master of Puppets
● Puppet master
• CA authority
• Modules
• Node descriptions
• Compare, compile, apply
● Master is not a requirement !

Puppet Clients

● Puppet client nodes
• Daemon
• Cron jobs
• External orchestration:
• for i in $hosts; do ssh $i “puppetd --test”; done
• mCollective, Func, …

Facts
● Facts
# facter

memoryfree => 387.21 MB
memorysize => 492.75 MB
swapfree => 481.00 MB
swapsize => 481.00 MB

domain => dev.inuits.be
fqdn => node3.dev.inuits.be
hostname => node3
interfaces => eth0
ipaddress => 172.16.142.141
macaddress => 00:0c:29:42:0b:8a
netmask => 255.255.255.0

Modules
● Dedicated per service
● Reusable
● Called from the manifests
● Live in /etc/puppet/modules/

Module Structure
● Files
● Templates
• Dynamic content
• Variables
<IfModule mpm_worker_module>
StartServers <%= StartServers %>
MaxClients <%= MaxClients %>
MinSpareThreads <%= MinSpare %>
MaxSpareThreads <%= MaxSpare %>
ThreadsPerChild <%= ThreadsChild %>
MaxRequestsPerChild <%= RequestsChild %>
</IfModule>
● Manifests

Modules
● Files
● Templates
● Manifests
• DSL
• Classes
• Elements

Node definitions
● Nodes.pp
class defaults {
$search = "inuits.be"
$nameservers = ['208.67.220.220', '208.67.222.222']

include dns::resolv
include ssh::keys
include ssh::server
}

node "ns1.dev.inuits.be" {
include defaults
include dns::powerdns::server
include dns::powerdns::resolver
}

node “web1.dev.inuits.be” {
include defaults
include apache2
include mysql
}

Ralsh
● Simplifies writing manifests
● Will generate parts of the manifest for you
● Based on your running config
● Limited functionality
master1.dev.inuits.be:~# ralsh user root
user { 'root':
uid => '0',
gid => '0',
comment => 'root',
ensure => 'present',
password => 'f34wi94$PmlI0CxQLb9HD',
shell => '/bin/bash',
home => '/root'
}

master1.dev.inuits.be:~# ralsh service apache2
service { 'apache2':
ensure => 'running',
enable => 'true'
}

Puppetizing your Infra
● Define common parts
● Define unique parts
● Write your manifests
● Use modules
• Puppet Forge
• GitHub
• Your own modules

What is sipXecs ?
● sipX ECS (Enterprise Communications Server)
● Open Source voice over IP telephony server
● Implementation of the Session Initiation Protocol (SIP)
● IP based communications system (IP PBX)
● Not unlike Asterisk
● Development started in 1999
● GNU Lesser General Public License (LGPL)
● Commercial offering from eZuce Inc.
● Designed around FreeSWITCH
● Modular and highly scalable system

We don't know VOIP
● External VOIP consultancy
• Hardware selection
• Codecs etc
• Scale out
● Irc.freenode.org #sipx

● s/don/didn/t
● Don't buy the book

Installing sipxecs
● Prebuilt ISO
● Kickstart
● Install scripts placed in .bashrc
● Ncurses based
● Lots of python scripts
● Heavy GUI usage

Why not Just ?
● Backup and Restore ?
• CDR Integration etc
● Image ?

● Productization
• Think 20-100 setups
• For different customers
• Different networks, different domains

So, that Python Script ?
● Configures your network
● Configures your dhcpd
● Configures your dns
● Configures your ntpd
● Configures your tftp
● Generates SSL stuff for you

There's puppet modules for that !

SipXconfig
● Is enabled by writing
“enabled” to /var/sipxdata/process-state/ConfigServer
● The configuration and management server (sipXconfig)
provides Web administration and user portals, Web services
APIs, as well as all the abstraction logic to make using
sipXecs as simple as it is. It provides centralized
management of all the aspects of sipXecs, including
installation, configuration, backup & restore, upgrade,
troubleshooting and cluster management.
● “Pushes” configs to other nodes
● Should be rewritten in Puppet or a like.

Configuring sipXecs
● A couple of files

● Some of them even obsoleted
● Putting the SSL stuff in the right location

Everything is a funky SSL
problem
● Sipx generates keys at install time
• Ca + keypairs per node
● 2nd node needs those keys
● Copy to puppetmaster and transfer back to other nodes ?

● Or generate on puppetmaster and redistribute ?

=> Generated on Puppetmaster

Adding a second node
● <> clustering
● <> high availability ( please don't start crying)

● Create an entry in the management interface
● Then repeat manual installation using ncurses

● Or just do a wget to register it with the primary

class voip::sipx {
sipx::netconfig {
"sipx":
ipaddress => $ip_address,
netmask => $netmask;
}
if $nodename == 'sipx-a' {
sipx::configserver{ "sipx": }
sipx::staticcertdbca{ "$hostname": }
sipx::staticcertdbnodes{ "SIPX-A.${platformdomainextension}":
clientname => "SIPX-A"; }
sipx::staticcertdbnodes{ "SIPX-B.${platformdomainextension}":
clientname => "SIPX-B"; }
include sipx::runmaster
}
else {
include sipx::runslave
sipx::register{ "$nodename":
clientname =>"${nodename}.${platformdomainextension}",
password =>"yourpw",}
}
sipx::supervisor { "$hostname":
sipx_supervisor => "sipx-a.$platformdomainextension";
}
sipx::staticssl{ "$hostname": }
}

More complexity
Or regular puppet ordering

● Sipx requires PgSQL
● You want PgSQL on an isolated LV
● PgSQL configuration has to be done after it initialized a DB
● SipX insist on starting PgSQL for you

class voip::storage {
file {
"/var/lib/pgsql":
ensure => directory;
lvm::volume { "pgsql":
vg => "systemvg",
pv => "/dev/cciss/c0d0p2",
fstype => "ext3",
size => "20G",
ensure => present,
}
mount { "/var/lib/pgsql":
atboot => true,
device => "/dev/systemvg/pgsql",
ensure => mounted,
fstype => "ext3",
options => "defaults",
require => [Logical_volume['pgsql'],File['/var/lib/pgsql']],
}
}
class voip::pgsql {
include postgres
postgres::initdb { "sipx": }
postgres::config{ "sipx":
listen => "*",
postgres::hba { "sipx":
allowedrules => [
"host SIPXCDR all ${clientip}/32 trust",
],
}
}

include voip::storage

include voip::pgsql

include voip::sipx

Class["voip::storage"] -> Class["voip::pgsql"] -> Class["voip::sipx"]

More complexity
Or manipulating content of files

● Bug in SIPX writes wrong listen adress for Freeswitch

● XML Gibberish
● Sipxconfig generates parts of the config files from unknown
sources

Augeas
With Augtool

augtool > set /augeas/load/Xml/incl[3] /tmp/blah.xml
augtool > set /augeas/load/Xml/lens Xml.lns
augtool > load
augtool >print /files/tmp/blah.xml/profile/settings/param[17]/
/files/tmp/blah.xml/profile/settings/param[17] = "#empty"
/files/tmp/blah.xml/profile/settings/param[17]/#attribute
/files/tmp/blah.xml/profile/settings/param[17]/#attribute/name = "sip-ip"
/files/tmp/blah.xml/profile/settings/param[17]/#attribute/value = "10.255.202.90"
augtool> print /files/tmp/blah.xml/profile/settings/param[18]/
/files/tmp/blah.xml/profile/settings/param[18]/#attribute/name = "ext-rtp-ip"
/files/tmp/blah.xml/profile/settings/param[18]/#attribute/value = "auto-nat"
augtool> print /files/tmp/blah.xml/profile/settings/param[16]/
/files/tmp/blah.xml/profile/settings/param[16]/#attribute/name = "rtp-ip"
/files/tmp/blah.xml/profile/settings/param[16]/#attribute/value = "10.255.202.90"

Augeas
augtool> get
/files/etc/sipxpbx/freeswitch/conf/sip_profiles/sipX_profile.xml/profile/settings/param[17]/#attribut
e/value
/
files/etc/sipxpbx/freeswitch/conf/sip_profiles/sipX_profile.xml/profile/settings/param[17]/#attribute
/value = 10.255.202.90
augtool> set
/files/etc/sipxpbx/freeswitch/conf/sip_profiles/sipX_profile.xml/profile/settings/param[16]/#attribut
e/value 10.0.0.2

Augeas & Puppet
augeas{"sipxprofile" :
changes => [
"set /augeas/load/Xml/incl[last()+1]/etc/sipxpbx/freeswitch/conf/sip_profiles/sipX_profile.xml",
"set /files/etc/sipxpbx/freeswitch/conf/sip_profiles/sipX_profile.xml/profile/settings/param[16]/#attribute/value 10.0.0.2",
"set /files/etc/sipxpbx/freeswitch/conf/sip_profiles/sipX_profile.xml/profile/settings/param[17]/#attribute/value 10.0.0.2",
],
}

augeas{"sipxprofile" :
lens => "Xml.lns",
incl => "/etc/sipxpbx/freeswitch/conf/sip_profiles/sipX_profile.xml",
context => "/files/etc/sipxpbx/freeswitch/conf/sip_profiles/sipX_profile.xml",
changes => [
"set profile/settings/param[16]/#attribute/value $ipaddress",
"set profile/settings/param[17]/#attribute/value $ipaddress",
],
onlyif => "get profile/settings/param[16]/#attribute/value != $ipaddress",
}

Manual config of the
services via the gui is still
required :(

I want to
● Automatically create my admin pw
● Automatically add that second node
● Automatically disable/ enable functions in the sipX server
• e.g conferencing, openfire
● Add users/phones

● There's an API !
● Which only implements limited functionality , and no
configuration

Screen scraping ?
(03:28:30 PM) lazyboy: y, you just need a form processing library, one that can read a form
values and allow you to post back your changes

(03:30:04 PM) lazyboy: the problem w/this method as you know is that it is constantly
breaking

(03:30:41 PM) sdog: yep .. whan you change the gui .. it will break ....

(03:30:45 PM) lazyboy: maybe we need a serverside abstraction layer, that does the
screenscraping and exports out a clean REST API

(03:31:13 PM) lazyboy: overtime, APIs go straight thru

(03:36:18 PM) lazyboy: so it's possible some of what you want to do is available w/not a lot
of screen scraping.

Abusing Test Frameworks to
configure services on a
webgui

Cucumber
● Looks extremely easy
• “Hey our manager could write these test”
● Isn't
• Heavily under documented
• Best docs are in the RSpec book
• Online examples are mostly broken
● Requires to write a lot of code

Apache Jmeter
● Test tool
● Load generation tool
● Lets you record session by
using a proxy
● Only recent versions support
SSL

Selenium
● Firefox plugin
● Replays your actions
• No need to write code
● Can export to perl, php,
ruby ..
• Which requires the a
Selenium Remote Control
Server
• Which launches Firefox
● SSL Fun ahead

Alternatives
● Sahi
• Similar to selenium
• Requires proxy
● www::mechanize
● Mechanize rubygem
● Webtest
● Your idea ?

Conclusions
● No good solution yet :(
● Talk to your upstream supplier
• Vendor / project
● Be patient
● Show the good example
● All bugs produced during this experience are on
https://github.com/KrisBuytaert

Contact
Kris Buytaert
Kris.Buytaert@inuits.be

Further Reading
@krisbuytaert
http://www.krisbuytaert.be/blog/
http://www.inuits.be/

Inuits Esquimaux
't Hemeltje Kheops Business
Gemeentepark 2 Center
2930 Brasschaat Avenque Georges
891.514.231 Lemaître 54
6041 Gosselies
+32 473 441 636 889.780.406

Automating complex infrastructures with Puppet

More Related Content

What's hot (20)

Similar to Automating complex infrastructures with Puppet (20)

More from Kris Buytaert (20)

Recently uploaded (20)

Automating complex infrastructures with Puppet