London Hashicorp Meetup #8 - Testing Programmable Infrastructure By Matt Long
14 likes3,988 views
This document discusses testing programmable infrastructure. It describes testing different aspects of infrastructure including a cloud broker and Kubernetes cluster. It recommends applying software development testing practices like unit testing, integration testing, and acceptance testing to infrastructure code and configuration. Various tools are presented for different types of infrastructure testing including Serverspec, Bats, Goss, Terraform_validate, and using Test Kitchen with Inspec. The document emphasizes that testing is important for infrastructure but often overlooked, and that testers and operations teams should work together to address new challenges in testing programmable systems.
![TESTING PROGRAMMABLE INFRASTRUCTURE
UNDER THE CUCUMBER, PLAIN RUBY
Then(/^I should be able to echo "([^"]*)"$/) do |text|
cmd = "echo #{text}"
output = @session.exec!(cmd)
close_ssh(@session)
expect(output.to_s.strip).to eql(text)
end](https://image.slidesharecdn.com/stablefoundations3-170217101103/85/London-Hashicorp-Meetup-8-Testing-Programmable-Infrastructure-By-Matt-Long-60-320.jpg)
![TESTING PROGRAMMABLE INFRASTRUCTURE
TestCreate Config Destroy
]TEST KITCHEN MANAGES YOUR TEST LIFECYCLE](https://image.slidesharecdn.com/stablefoundations3-170217101103/85/London-Hashicorp-Meetup-8-Testing-Programmable-Infrastructure-By-Matt-Long-84-320.jpg)
London Hashicorp Meetup #8 - Testing Programmable Infrastructure By Matt Long
- 2. PROGRAMMABLE INFRASTRUCTURE IS GREAT, BUT WE'RE MISSING SOMETHING. TESTING.
- 3. I'M A TESTER HELLO, I'M MATT
- 4. I WORK HERE ↑
- 5. I AM NOT A SYSADMIN
- 7. TESTING PROGRAMMABLE INFRASTRUCTURE THE APPLICATION OF METHODS AND TOOLING FROM SOFTWARE DEVELOPMENT TO MANAGEMENT OF IT INFRASTRUCTURE PROGRAMMABLE INFRASTRUCTURE IS.. THE INTERNET
- 8. TESTING PROGRAMMABLE INFRASTRUCTURE EXAMPLES OF PROGRAMMABLE INFRASTRUCTURE ▸ Automated provisioning & configuration ▸ Configuration as code ▸ Version / source controlled
- 9. TESTING PROGRAMMABLE INFRASTRUCTURE TOOLING EXAMPLES
- 10. PROGRAMMABLE INFRASTRUCTURE IS AWESOME! Credit: Vault Boy, Bethesda Softworks
- 11. IT'S FAST!
- 12. IT'S AUTOMATIC!
- 13. IT'S ALL CODE!
- 15. TESTING IS USED TO MITIGATE COMPLEXITY & RISK
- 16. BUT TESTING IS RARE Credit: Gunshow, KC Green
- 17. TESTING PROGRAMMABLE INFRASTRUCTURE WHAT I'M GOING TO TALK ABOUT ▸PART 1: Testing a cloud broker ▸PART 2: Building a Kubernetes cluster ▸CONCLUSIONS
- 18. TESTING A CLOUD BROKER AN INFRASTRUCTURE HEAVY PRODUCT
- 19. THE PROBLEM
- 20. TESTING PROGRAMMABLE INFRASTRUCTURE WE WANT TO MOVE TO THE CLOUD... BUT WE'RE WARY OF LOCK IN Large organisation
- 21. TESTING PROGRAMMABLE INFRASTRUCTURE USERS USE MULTIPLE CLOUD PROVIDERS
- 22. TESTING PROGRAMMABLE INFRASTRUCTURE PROBLEMS ▸ Different interfaces, feature sets & lingo ▸ Can't switch easily ▸ Spending difficult to track ▸ Temptation to fall back on most popular
- 24. TESTING PROGRAMMABLE INFRASTRUCTURE BENEFITS ▸ Quick, easy provisioning ▸ one team previously took 3 months ▸ Common interface to cloud features ▸ Templates for common dev environments ▸ Built in best practice: monitoring, security ▸ Track spending
- 25. THIS IS A REALLY COMPLICATED APPLICATION
- 27. TESTING PROGRAMMABLE INFRASTRUCTURE WORKFLOW ▸ Log into Web UI ▸ Fill in information about environment ▸ Broker creates and bootstraps resources ▸ SSH into resources
- 28. TESTING PROGRAMMABLE INFRASTRUCTURE WEB TESTING ▸ Log into Web UI ▸ Fill in information about environment
- 29. TESTING PROGRAMMABLE INFRASTRUCTURE ??? ▸ Broker creates and bootstraps resources ▸ SSH into resources
- 30. HOW DO YOU TEST INFRASTRUCTURE?
- 31. TESTING PROGRAMMABLE INFRASTRUCTURE WHAT TO TEST? Do our deployment scripts work? Does the VPN server work? Can instances access one another? Are services running? Can I SSH into a server?
- 33. TESTING PROGRAMMABLE INFRASTRUCTURE Does the VPN box work? Can I SSH into a server? Do our deployment scripts work? Are services running? ANOTHER TESTING PYRAMID? credit: Ubuntu dev quality guide https://developer.ubuntu.com/en/phone/platform/quality/ Can instances access one another?
- 34. TOOLING
- 35. TESTING PROGRAMMABLE INFRASTRUCTURE TOOLS AVAILABLE ▸ Bats ▸ ShUnit2 ▸ Goss ▸ ServerSpec / Inspec / TestInfra ▸ Test Kitchen
- 36. UNIT TESTING
- 37. TESTING PROGRAMMABLE INFRASTRUCTURE BATS ▸ "Bash Automated Testing System" ▸ Unit testing for bash ▸ Like JUnit
- 38. TESTING PROGRAMMABLE INFRASTRUCTURE SH UNIT 2 ▸ Shell unit testing framework ▸ Runs on all Bourne shells ▸ sh, BASH, DASH, ksh, zsh ▸ No activity or support?
- 39. INTEGRATION TESTING OR: SERVER VALIDATION
- 40. TESTING PROGRAMMABLE INFRASTRUCTURE GOSS ▸ Go based ▸ Specs in YAML ▸ Minimal, fast, and simple ▸ Some neat features ▸ .. have to run on the server ▸ .. no Windows support
- 41. TESTING PROGRAMMABLE INFRASTRUCTURE SERVERSPEC ▸ Server based assertions ▸ Ruby/RSpec based ▸ Probably the most famous ▸ Can SSH into instances
- 42. TESTING PROGRAMMABLE INFRASTRUCTURE INSPEC ▸ Written & maintained by Chef ▸ Very similar to ServerSpec ▸ Different feature set ▸ More focused on compliance
- 43. TESTING PROGRAMMABLE INFRASTRUCTURE TESTINFRA ▸ ServerSpec, but in Python
- 44. TEST HARNESS
- 45. TESTING PROGRAMMABLE INFRASTRUCTURE TEST KITCHEN ▸ Orchestrates setup, test, teardown ▸ Runs BATS, shUnit2, RSpec, Serverspec ▸ Popular in the Chef community ▸ Not suitable for our cloud broker
- 46. OUR SOLUTION
- 48. TESTING PROGRAMMABLE INFRASTRUCTURE USERS WEB TEST FRAMEWORK
- 49. TESTING PROGRAMMABLE INFRASTRUCTURE USERS INFRASTRUCTURE TEST FRAMEWORK
- 50. TESTING PROGRAMMABLE INFRASTRUCTURE USERS WEB TESTS https://github.com/opencredo/test-automation-quickstart
- 51. TESTING PROGRAMMABLE INFRASTRUCTURE INFRASTRUCTURE TESTS Serverspec
- 52. TESTING PROGRAMMABLE INFRASTRUCTURE INFRASTRUCTURE TESTING STACK / Serverspec ???
- 53. TESTING PROGRAMMABLE INFRASTRUCTURE WHY RUBY? ▸ Fantastic testing community ▸ More suitable for SSHing into boxes ▸ "Win RM" gem ▸ Ops already familiar with it ▸ Reduces tech stack
- 54. TESTING PROGRAMMABLE INFRASTRUCTURE SERVERSPEC SMOKE TESTS ▸ Run before everything else ▸ Really quick ▸ Catches obvious errors ▸ Not complex tasks
- 55. TESTING PROGRAMMABLE INFRASTRUCTURE SERVERSPEC EXAMPLE describe package('jenkins') do it { should be_installed } end describe service('jenkins') do it { should be_enabled } it { should be_running } end describe port(8080) do it { should be_listening } end
- 56. TESTING PROGRAMMABLE INFRASTRUCTURE Background: Given environment has been created And the following user details: | user_alias | username | public_key | | userA | envoy | test | Scenario: IPA - Login via SSH Key authentication succeeds Given user "userA" is authorised to access environment vms When user "userA" starts ssh session in host "env" Then I should be able to echo "hello world" CUCUMBER FOR ACCEPTANCE TESTING
- 57. TESTING PROGRAMMABLE INFRASTRUCTURE Background: Given environment has been created And the following user details: | user_alias | username | public_key | | userA | envoy | test | Scenario: IPA - Login via SSH Key authentication succeeds Given user "userA" is authorised to access environment vms When user "userA" starts ssh session in host "env" Then I should be able to echo "hello world" CUCUMBER FOR ACCEPTANCE TESTING Cloud broker APIs
- 58. TESTING PROGRAMMABLE INFRASTRUCTURE Background: Given environment has been created And the following user details: | user_alias | username | public_key | | userA | envoy | test | Scenario: IPA - Login via SSH Key authentication succeeds Given user "userA" is authorised to access environment vms When user "userA" starts ssh session in host "env" Then I should be able to echo "hello world" CUCUMBER FOR ACCEPTANCE TESTING Standard Ruby
- 59. TESTING PROGRAMMABLE INFRASTRUCTURE Background: Given environment has been created And the following user details: | user_alias | username | public_key | | userA | envoy | test | Scenario: IPA - Login via SSH Key authentication succeeds Given user "userA" is authorised to access environment vms When user "userA" starts ssh session in host "env" Then I should be able to echo "hello world" CUCUMBER FOR ACCEPTANCE TESTING RSpec assertions
- 60. TESTING PROGRAMMABLE INFRASTRUCTURE UNDER THE CUCUMBER, PLAIN RUBY Then(/^I should be able to echo "([^"]*)"$/) do |text| cmd = "echo #{text}" output = @session.exec!(cmd) close_ssh(@session) expect(output.to_s.strip).to eql(text) end
- 61. THOUGHTS
- 62. TESTING PROGRAMMABLE INFRASTRUCTURE THE GOOD ▸ Specialised tests for each layer ▸ Really quick, expressive ServerSpec tests ▸ Power of a full programming language for user tests
- 63. TESTING PROGRAMMABLE INFRASTRUCTURE THE BAD ▸ Over reliance on acceptance tests ▸ Awkward switching between two suites ▸ Out of my comfort zone
- 64. TESTING PROGRAMMABLE INFRASTRUCTURE THE UGLY ▸ Starting infrastructure is SLOW. ▸ It's expensive...
- 65. IT WAS WORTH IT DESPITE ALL THAT
- 66. BUILDING A KUBERNETES CLUSTER APPLYING TDD TO INFRASTRUCTURE
- 68. I LEARNED A LOT! Credit: The Simpsons, Fox
- 69. TESTING PROGRAMMABLE INFRASTRUCTURE BUILD THIS: WITH THESE:
- 70. TESTING PROGRAMMABLE INFRASTRUCTURE NOT A STRAIGHTFORWARD TASK
- 71. TESTING PROGRAMMABLE INFRASTRUCTURE BUT HOW TO TEST IT? ▸ This is a dev activity ▸ Want fast feedback ▸ Complexity is mitigated by testing!
- 72. TESTING PROGRAMMABLE INFRASTRUCTURE ▸ Provisions cloud infrastructure ▸ Declarative files ▸ Some support for variables TERRAFORM
- 73. TESTING PROGRAMMABLE INFRASTRUCTURE TERRAFORM COMMANDS ▸ terraform plan ▸ Tells you what will change ▸ terraform apply ▸ Applies changes ▸ terraform validate ▸ Lints terraform syntax
- 74. TESTING PROGRAMMABLE INFRASTRUCTURE TERRAFORM FILE EXAMPLE resource "aws_instance" "etcd-node" { count = 3 ami = "ami-7abd0209" # centos availability_zone = "eu-west-1a" # ireland instance_type = "t2.micro" subnet_id = .... private_ip = .... key_name = "${aws_key_pair.my-key.key_name}" }
- 75. TESTING PROGRAMMABLE INFRASTRUCTURE LINT WITH 'TERRAFORM VALIDATE' COMMAND Omitting a variable:
- 76. TESTING PROGRAMMABLE INFRASTRUCTURE BUT IT DOESN'T CATCH ALL PROBLEMS Duplicate subnet CIDRS:
- 77. TESTING PROGRAMMABLE INFRASTRUCTURE LINTING ISN'T ENOUGH ▸ Devs don’t just rely on compilers ▸ We need something more powerful Credit: Nick Cave, "Soundsuit"
- 78. UNIT TESTING
- 79. TESTING PROGRAMMABLE INFRASTRUCTURE TERRAFORM_VALIDATE ▸ Python based unit testing ▸ NOT to be confused with 'validate' command ▸ Builds map of resources & properties ▸ Totally offline ▸ New and incomplete https://github.com/elmundio87/terraform_validate
- 80. TESTING PROGRAMMABLE INFRASTRUCTURE TERRAFORM_VALIDATE FORK OC has forked the terraform validate repo https://github.com/opencredo/terraform_validate
- 82. TESTING PROGRAMMABLE INFRASTRUCTURE GOSS ▸ Easy to get up and running ▸ Doesn’t support remote # example usage: ./goss-test.sh 34.248.91.167 TARGET='centos@'$1 SSH_KEY_PATH=~/.ssh/aws ssh -t -t -i $SSH_KEY_PATH $TARGET 'curl -fsSL https://goss.rocks/install | sudo sh' scp ./goss.json $TARGET:~/goss.yaml ssh -t -t -i $SSH_KEY_PATH $TARGET 'goss validate' https://gist.github.com/burythehammer/081d6ee11cc33c2f4c4729ae67622f5b
- 83. TESTING PROGRAMMABLE INFRASTRUCTURE ▸ Terraform compatibility ▸ Already a talk about this ▸ “Untangling Infrastructure Code” by Nell Shamrell-Harrington TEST KITCHEN + INSPEC
- 84. TESTING PROGRAMMABLE INFRASTRUCTURE TestCreate Config Destroy ]TEST KITCHEN MANAGES YOUR TEST LIFECYCLE
- 85. TESTING PROGRAMMABLE INFRASTRUCTURE TestCreate Config Destroy TEST KITCHEN DOESN'T SUPPORT MULTIPLE PROVISIONERS
- 86. TESTING PROGRAMMABLE INFRASTRUCTURE TEST KITCHEN DOESN'T SUPPORT MULTIPLE PROVISIONERS https://github.com/test-kitchen/test-kitchen/issues/329
- 87. TESTING PROGRAMMABLE INFRASTRUCTURE TERRAFORM 'NULL RESOURCE' resource "null_resource" "ansible" { triggers { instance_ids = "${join(",", aws_instance.etcd-node.*.id)}" } provisioner "local-exec" { command = "sleep 20 && cd ../ansible/ && ansible-playbook etcd.yaml" } }
- 89. THOUGHTS
- 90. TESTING PROGRAMMABLE INFRASTRUCTURE THE GOOD ▸ Tooling exists! ▸ You can totally get a test suite working Credit: Overwatch, Blizzard Entertainment
- 91. TESTING PROGRAMMABLE INFRASTRUCTURE THE BAD ▸ Unit testing extremely immature ▸ Tools immature in general Credit: Overwatch, Blizzard Entertainment
- 92. TESTING PROGRAMMABLE INFRASTRUCTURE THE HACKY ▸ Be prepared to hack ▸ It might not even be possible Credit: Overwatch, Blizzard Entertainment
- 93. THIS IS BRAND NEW GROUND REMEMBER:
- 94. TESTING TOOLS DEPEND ON YOUR STACK
- 95. CONCLUSIONS
- 96. TESTING IS IMPORTANT BUT OFTEN IGNORED
- 97. TESTERS AND OPS SHOULD WORK TOGETHER WE NEED TO GET OUT OF OUR COMFORT ZONES
- 98. TOOLS EXIST BUT BE PREPARED TO HACK
- 99. FINALLY...
- 100. TESTING PROGRAMMABLE INFRASTRUCTURE THE APPLICATION OF METHODS AND TOOLING FROM SOFTWARE DEVELOPMENT TO MANAGEMENT OF IT INFRASTRUCTURE PROGRAMMABLE INFRASTRUCTURE IS..
- 101. TESTING IS A SOFTWARE DEVELOPMENT METHOD WE SHOULD APPLY IT TO INFRASTRUCTURE