Assurer - a pluggable server testing/monitoring framework

Assurer – A server testing/monitoring framework Gosuke Miyashita http://mizzy.org/

Myself Known as Chocoboy Miyashita somewhere Working in paperboy&co. Mainly developed public/private web service APIs with perl and catalyt ... but now also woking on servers and networks and so on Today’s my son’s entrance ceremony of an elementary school

What’s Assurer ? A server testing/monitoring framework like Plagger Plugin mechanism YAML configurable assets Test by Test::Base Originally developed for one time test of servers that are newly built Monitoring seems like testing, so decided to cover monitoring

Where the name Assurer derived from ? Pronounce “A-shu-ra” Plagger like tools ended with “er” Archer by tokuhirom Observer by Hatena (not in public) Dishuber by Yappo Precure by Kan Fushihara Testing is also called “Quality Assurance” MISS

Execution phases of Assurer Test phase Execute test Nofity phase Notify test results Format phase Format test results Publish phase Output formatted test results

Relation of each phases Test Publish Format Nofity

Test phase Execute test Plugins on this phase: Test::HTTP Test::SMTP Test::DBI Test::Ping etc

Config of test phase test: - module: HTTP config: host: www.mizzy.org content: It works!

Notify フェーズ Notify test results Plugin on this phase: Notify::IRC

Config of notify phase notify: - module: IRC config: daemon_port: 9991 nickname: assurerbot server_host: chat.freenode.net server_port: 6667 server_channels: - #assurer - #assurer-jp charset: iso-2022-jp announce: notice

Format phase Format test results Plugins on this phase: Format::Text Fomat::HTML

Config of format phase format: - module: HTML config: css: /css/style.css

Publish phase Output formatted results Plugins on this phase: Publish::Term Publish::Mail Publish::File

Config of publish phase publish: - module: Mail config: subject: Test results from Assurer to: someone@example.com from: root@example.com publish: - module: Mail config: subject: Test results from Assurer to: someone@example.com from: root@example.com

Summary for now Assurer is a serverPlagger Test, Notify, Format and Publish phases Easily change tests and outputs with plugins

Hosts and roles Write target hosts and roles in one place on config.yaml

Config without hosts and roles test: - module: HTTP config: host: www0.mizzy.org - module: HTTP config: host: www1.mizzy.org

Config with hosts and roles test: - module: HTTP role: web hosts: web: - www0.mizzy.org - www1.mizzy.org

Filter Filtering test results on format and notify phase Filtering formatted results on publish phase

Filter on format phase Only OK results are formatted as text format format: - module: Text filter: module: Status status: ok

Filter on notify phase Only NOK results are notified to IRC notify: - module: IRC filter: module: Status status: not ok

Filter on publish phase Text formatted results are displayed on a terminal,html formatted results are sent as a mail format: - module: Text - module: HTML pusblish: - module: Term filter: module: Type type: text/plain - module: Mail filter: module: Type type: text/html

Config validation Kwalify – YAML schema validator http://www.kuwata-lab.com/kwalify / Validate common config Validate config of each plugin Kwalify schemas for each plugin put under assets/kwalify/plugins

Mistake on common config test s : # test is right - module: Test::HTTP $ ./assurer.pl -c config.yaml Assurer::ConfigLoader [fatal] - [/] Expected required key `test‘ - [/tests] Unexpected key `tests' at line 46

Mistake on plugin config test: - module: HTTP config: content s : XXX # content is right $ ./assurer.pl -c config.yaml Assurer::ConfigLoader [fatal] Config error in Test::HTTP - [/contents] Unexpected key `contents' at line 66

Shell mode Execute shell command on hosts in config.yaml

Shell mode $ ./assurer.pl --shell assurer> uptime [www.mizzy.org] 21:48:24 up 5 days, 23:08, 2 users, load average: 0.24, 0.16, 0.17 [svn.mizzy.org] 21:48:24 up 5 days, 23:08, 2 users, load average: 0.24, 0.16, 0.17 [ftp.mizzy.org] 21:48:24 up 5 days, 23:08, 2 users, load average: 0.24, 0.16, 0.17

Exec on hosts belong to specified role $ ./assurer.pl --shell --role=web assurer> uptime [www.mizzy.org] 21:48:24 up 5 days, 23:08, 2 users, load average: 0.24, 0.16, 0.17

Special command !on assurer> !on app1.foo.com \ app2.foo.com do uptime # Exec on app1.foo.com and app2.foo.com only assurer> !on /.*\.foo\.com/ do \ uptime # Exec on hosts matched .*\.foo\.com only

Special command !with assurer> !with web db do uptime # exec on hosts belong to web role and db role assurer> !with /web|mail/ do uptime # exec on hosts belong to role match web or mail

Special command !test assurer> !test HTTP Assurer::Plugin::Test::HTTP [info] Testing HTTP on www0.mizzy.org ... ok 1 - HTTP status code of http://mizzy.org:80 is 200 not ok 2 – Content of http://mizzy.org:80 matches 'It works!‘ assurer>

Special command !test assurer> !test HTTP on app1.foo.com # test http on app1.foo.com assurer> !test HTTP on /.*.foo.com/ # can use regexp assurer> !test HTTP with web # test http on hosts belong to web role assurer> !test HTTP with /web|mail/ # can use regexp

Pallarel testing Multiple test processes on test phase Default is 5 Can change number processes working at once $ assurer.pl --para=20

Distributed testing localhost test0.mizzy.org www0.mizzy.org www1.mizzy.org www2.mizzy.org test1.mizzy.org test2.mizzy.org 1. ssh assurer_test.pl 3. Return result 2. Exec test

Config for distributed testing exec_on: - host: test0.mizzy.org priority: 3 - host: test1.mizzy.org priority: 2 - host: test2.mizzy.org priority: 1

Auto config generation Scan hosts in template.yaml by Nmap::Scanner Detect services working on the hosts and generete config.yaml automatically config templates for each plugin put under assets/discover (http.yaml, mysql.yaml, etc) # assurer.pl --discover -c template.yaml

Integragte Test::WWW::Mechanize

Integrate Test::WWW::Mechanize Can write tests using Test::WWW::Mechanize Put .pl files under assets/plugins/Test-WWW-Mechanize Pre-set $mech, $host on execution

asset file for Test::WWW::Mechanize # $host and $mech are set by Assure core $mech ->get_ok("http:// $host ", "got htttp:// $host "); $mech ->content_contains('It works!', "Content matches 'It works!'");

Assurer::Test module Export Test::More like methods is,() like(), ok() Output test results formatted as TAP (Test Analyze Protocol)

Inside test plugins package Assurer::Plugin::Test::HTTP; use base qw( Assurer::Plugin::Test ); use Assurer::Test; sub register { my $self = shift; $self->register_tests( qw/ status content server / ); } sub status { my ( $self, $context, $args ) = @_; ... 中略 is( $res->code, '200', 'HTTP status code of $self->{url} is 200' ); }

Internal mechanism in test phase

Internal mechanism in test phase assurer.pl Assurer::bootstrap() Assurer::Dispatch::run() call exec directly or by ssh exec tests return test results call generate result objects next phase assurer_test.pl assurer_test.pl assurer_test.pl assurer_test.pl assurer_test.pl Assurer::Result Assurer::Result Assurer::Result Assurer::Result Assurer::Result

Assurer::Dispatch::run() Exec assurer_test.pl for each test Pallarelize by POE::Wheel::Run Number of processed controlled by POE::Component::JobQueue Pass config for plugin and context object serialized by YAML and encoded by Base64

assurer_test.pl assurer_test.pl -– config = LS0tCmNvbmZpZzoKICBjb250ZW50OiBJ dCB3b3JrcyEKICBob3N0OiB3d3cubWl6enkub3JnCm1vZHVsZTogSFRUUApuYW1lOiBIVFRQIHRlc3QKcm9sZTogd2ViCg== - - context =LS0tICEhcGVybC9oYXNoOkFzc3VyZXIKYmFzZV9kaXI6IC9 ob21lL21peWEvc3ZrL0Fzc3VyZXIKY29uZmlnOgogIGZvcm1hdDoKICAgIC0gY29uZmlnOiAmMSB7fQogICAgICBtb2R1bGU6IFRleHQKICBnbG9iYWw6CiAgICBob3N0OiB+CiAgICBpbnRlcnZhbDogMwogICAgbG9nOgogICAgICBsZXZlbDogZGVidWcKICAgIG5vX2RpYWc6IDAKICAgIHJldHJ5OiAzCiAgaG9zdHM6CiAgICBmdHA6CiAgICAgIC0gZnRwLm1penp5Lm9yZwogICAgc3ZuOgogICAgICAtIHN2bi5taXp6eS5vcmcKICAgIHdlYjoKICAgICAgLSB3d3cubWl6enkub3JnCiAgICAgIC0gc3ZuLm1penp5Lm9yZwogICAgICAtIHRyYWMubWl6enkub3JnCiAgbm90aWZ5OgogICAgLSBjb25maWc6CiAgICAgICAgYW5ub3VuY2U6IG5vdGljZQogICAgICAgIGNoYXJzZXQ6IGlzby0yMDIyLWpwCiAgICAgICAgZGFlbW9uX3BvcnQ6IDk5OTEKICAgICAgICBuaWNrbmFtZTogYXNzdXJlcmJvdAog ... cont ...

Why POE::Wheel::Run ? Would like to exec tests as other processes Need non-blocking I/O with single process and POE Can easily distribute to remote servers (exec directly on local, exec by ssh on remote ） Communication with test processes Can communicate easily with test processes by STDIN, STDOUT and STDERR

Roadmap Integrate with Nagios/NRPE (Test::Nagios) Enhance notify phase Store Plugin (store test results in database) Integrate with Archer, puppet (share host lists) Test of Assurer itsself

Web sites and IRC channels http://assurer.jp/trac http://mizzy.org/ #assurer@chat.freenode.net #assurer-ja@chat.freenode.net

Assurer - a pluggable server testing/monitoring framework

More Related Content

What's hot (20)

Viewers also liked (6)

Similar to Assurer - a pluggable server testing/monitoring framework (20)

More from Gosuke Miyashita (20)

Recently uploaded (20)

Assurer - a pluggable server testing/monitoring framework