Introduction to Chef: Automate Your Infrastructure by Modeling It In Code
23 likes16,618 views
The document introduces Chef, a configuration management tool that automates server management by defining the desired state of servers through code. It outlines the challenges of traditional server management and explains the architecture of Chef, including the roles of nodes, management servers, and cookbooks. Additionally, it provides insights into setup, key concepts, and resources for learning more about Chef.
![Roles
• The Run-List is usually the same for all servers at the
same “layer” in the stack and different across layers.
Runlist[Apt, Sample, Apache]
Runlist[Apt, Java, Tomcat, JBoss]
Runlist[Apt, Postgres, Newrelic]
“web” Role
“app” Role
“db” Role](https://image.slidesharecdn.com/chefkeynote-140405144556-phpapp01/85/Introduction-to-Chef-Automate-Your-Infrastructure-by-Modeling-It-In-Code-74-320.jpg)
![Tips for Setting Up Knife
• When you run knife configure --initial use your local paths for the admin.pem and the chef-
validator.pem
• At some point, you'll need to download files from /etc/chef-server folder on the Chef Server in order to
get Knife up and running.
• You'll need to modify your knife.rb file (e.g. to point to your cookbooks path) to get things working
right. Errors caused by this are not well documented. Here's my knife.rb:
Joshs-MacBook-Pro:.chef josh$ vim knife.rb !
!
log_level :info!
log_location STDOUT!
node_name 'josh'!
client_key '/Users/josh/.chef/josh.pem'!
validation_client_name 'chef-validator'!
validation_key '/repos/chef-repo/.chef/chef-validator.pem'!
chef_server_url 'https://chef.projname.mybiz.com'!
syntax_check_cache_path '/Users/josh/.chef/syntax_check_cache'!
cookbook_path '/repos/chef-repo/cookbooks'!
knife[:aws_access_key_id] = "Your AWS Access Key ID"!
knife[:aws_secret_access_key] = "Your AWS Secret Access Key"!
knife[:region] = "us-west-2"!
knife[:vpc_id] = "vpc-XXXXXXX"](https://image.slidesharecdn.com/chefkeynote-140405144556-phpapp01/85/Introduction-to-Chef-Automate-Your-Infrastructure-by-Modeling-It-In-Code-92-320.jpg)
Introduction to Chef: Automate Your Infrastructure by Modeling It In Code
- 1. Automate Your Server Configuration! by Modeling It In Code Josh Padnick Desert Code Camp April 5, 2014
- 2. Today’s 5-Course Meal • What is Chef and what does it solve? • Key Chef concepts • Where/How you use it • Tips & tricks • Learning more
- 3. Our Goal Today: You go home tonight and deploy your first Chef cookbook in 1 - 2 hours.
- 4. • Founder & CIO of Omedix • Special interest in Healthcare IT • 10+ years doing web app dev • Strong preference for open source & Java ecosystem Josh Padnick These slides are posted on http://joshpadnick.com
- 5. Some Problems from the Server Management World
- 6. “The Magic Server” The code doesn't work on any server except the magic server. We're afraid to touch it.
- 7. “Just Clone Another Instance” You can’t setup a dev machine quickly, but you're virtualizing and at least you can clone it.
- 8. “We have AMIs! But we need to change them :(“ You can clone a server, but your snapshot (AMI) needs to be changed...so you have to rebuild the AMI.
- 9. “I have a simple script to deploy! to 10 different servers” The script is small and simple, but deploying it to 10 servers is painful, so updates to this script will not be permitted!
- 10. Documentation is time-consuming and always out of date. But we still need to ramp up the new guy. “Documentation is critical…but out of date.”
- 11. "Deploying a new server takes too long" ! "This server here is broken so let's spend hours restoring it because launching a new server is too hard." ! "Changing the deployment process will take time we don't have" !
- 12. Are you actively managing more than a few servers on an ongoing basis? Welcome to! SERVER MANAGEMENT HELL
- 13. What is the underlying issue here? The real problem is STATE.
- 14. But wait, haven’t we dealt with state before? Only since 1937 when Alan Turing invented the state machine.
- 15. What does the Chef solution look like?
- 16. The Servers
- 18. The Servers Management Server This guy is the authority on what state each server should be in
- 19. The Servers Management Server Local Workstation
- 20. The Servers Each of these guys installs an agent called the CHEF CLIENT Management Server Local Workstation
- 21. The Servers Each of these guys installs an agent called the CHEF CLIENT Management Server Local Workstation This guy is the authority on what state each server should be in
- 22. The Servers Each of these guys installs an agent called the CHEF CLIENT Management Server Local Workstation This guy is the authority on what state each server should be in The DevOps engineer pushes all instructions to the Management Server
- 23. Now let’s use all the Chef lingo.
- 25. Nodes Chef Server Knife Write code that describes the state of a node and deploy it to Chef Server
- 26. Write code that describes the state of a node and deploy it to Chef Server Has the official record of what each server’s state SHOULD be Nodes Chef Server Knife
- 27. Each node periodically polls the Chef Server asking for “update state” instructions Has the official record of what each server’s state SHOULD be Nodes Chef Server Knife
- 28. Each node updates its state! based on instructions from the management server. Nodes Chef Server Knife
- 29. Nodes Chef Server Knife Each Node’s state has been updated!
- 30. Did someone say we’re writing code? • Write it in Ruby 2.x • Chef gives us a Ruby DSL specially for declaring server state • Version-control it with anything, but standard is git
- 31. We’ll go into more Chef detail later. Let’s pan out to the 50,000 foot view again.
- 32. Chef is considered a Configuration Management! Solution
- 33. What is Configuration Management? • A tool used to manage server configuration with automation. • Born out of the need for major websites (Amazon, Facebook, Yahoo) to manage huge numbers of servers.
- 34. Who Created Chef? Jesse Robbins Adam Jacob Managed lots of Amazon.com servers Real-life fire fighter! Built infrastructure for 15 companies Kept seeing the same patterns!
- 35. Alternatives to Chef • Ansible • Saltstack • Puppet
- 36. DevOps • Chef merges the worlds of Development (Dev) and SysAdmin (Ops). • Dev: build software, version control, automated testing • Ops: provisioning servers, maintaining servers, monitoring
- 37. Let’s talk more about key concepts in Chef.
- 38. • The primary currency in Chef is the Cookbook. • That’s the set of code and files we use to tell a server what state to update itself to.
- 39. • Sample cookbooks • MySQL • Nginx • Install Java • Update crontab • Update timezone • Add SSH certificates to authorized_certificates
- 40. • You can roll your own • But for almost every need, just search GitHub • google “github java cookbook” and choose the best one • Opscode has “Community Cookbooks at http:// community.opscode.com/cookbooks but I find it out of date and incomplete.
- 41. Sample Cookbook • Recipes are individual sets of instructions to be executed. • Recipes read values from Attributes for things like:! • Passwords • Filepaths • Usernames • Configuration options for applications
- 44. Attributes • How attributes get applied to a recipe are behind much of the perceived complexity of Chef. • We’ll cover this in more detail later.
- 45. Let’s see how a cookbook ultimately gets executed on a node.
- 46. Nodes Chef Server Knife At your workstation, download or write a new cookbook. Let’s call it “Sample”.
- 47. Nodes Chef Server Knife Upload your cookbook to the Chef Server using KNIFE.
- 48. Nodes Chef Server Knife Now has the latest version of the “Sample” cookbook.
- 49. Nodes Chef Server Knife Use KNIFE to tell CHEF SERVER the RUN-LIST for a specific node.
- 50. Nodes Chef Server Knife Now knows that Node 32 should run the “Sample” cookbook.
- 51. Nodes Chef Server Knife SSH directly into an individual node (“Node 32”) and run “sudo chef- client” to CONVERGE the node.
- 52. Nodes Chef Server Knife Node 32 uses chef-client to contact Chef Server. It get its updated run-list, and executes the run list.
- 53. Nodes Chef Server Knife Node 32’s run-list says to run the Sample Cookbook. It runs the latest version of Sample.
- 54. Nodes Chef Server Knife Node 32 has now executed all the instructions in the latest version of the “Sample” cookbook
- 55. Wait, we have to manually log into each node to update it? • No! In production, we use Roles to specify a run-list. • This way, we only update the Chef Server. Individual nodes poll the Chef Server every X minutes to check for updates.
- 56. Where Chef is Used
- 57. • Traditional Chef Server • AWS OpsWorks • Clould Management (Scalr & RightScale) • Vagrant
- 58. Traditional Chef Server Nodes Chef Server Knife
- 59. Traditional Chef Server • Chef Server comes in Open Source or Enterprise Edition • Chef Server is “on premise” or hosted by Opsworks
- 62. Different Flavors of Chef-Client • Server Mode • Chef Solo Mode • Local Mode
- 63. Cloud Management Services (e.g. Scalr & RightScale) Nodes Local Workstation
- 64. • It’s a command-line interface to virtual machine software like VirtualBox or VMWAre.
- 65. • You can combine Chef’s cookbooks, VirtualBox (a free VM provider), and Vagrant to run chef cookbooks directly on local VM’s! • The same cookbooks that define your infrastructure can now define your local dev environment.
- 66. • Get the latest cookbooks on your local machine • Run a bunch of VM’s with VirtualBox and Vagrant • Update the VM’s with Chef cookbooks Local Workstation
- 69. Remaining Concepts We’ll Talk About • Roles • Environments • Data Bags • Ohai • Source Control & Chef
- 70. Remaining Concepts We Won’t Cover • Resources • Providers • LWRPs
- 71. All those remaining concepts we’ll cover fall into two buckets: 1. Attributes 2. Run-lists
- 72. Attributes • Can be set in multiple places
- 73. Ohai • It’s a program that runs on each node and supplies attribute info specific to that node. • Examples • What OS the node is running • How much hard drive space • How much memory is available • Linux kernel version
- 74. Roles • The Run-List is usually the same for all servers at the same “layer” in the stack and different across layers. Runlist[Apt, Sample, Apache] Runlist[Apt, Java, Tomcat, JBoss] Runlist[Apt, Postgres, Newrelic] “web” Role “app” Role “db” Role
- 75. Roles • Roles can also specify attributes that OVERRIDE the “default” attributes set in the Recipe. • When a new node is created (“bootstrapped”) it is best practice to explicitly identify which Role it belongs to. • Roles are declared as a simple JSON file and uploaded to Chef Server using Knife.
- 77. Nodes Chef Server Knife Use KNIFE to bootstrap a new NODE with the “web” ROLE.
- 78. Nodes Chef Server Knife A new NODE exists that is assigned to the “web” role
- 79. Nodes Chef Server Knife Use KNIFE to update the “web” ROLE to add a new RECIPE to the RUN-LIST.
- 80. Nodes Chef Server Knife The “web” role has been updated.
- 81. Nodes Chef Server Knife New Node polls the CHEF SERVER every X minutes to look for updates.
- 82. Nodes Chef Server Knife The “web” ROLE now includes a new RECIPE so download this COOKBOOK and run the RECIPE
- 83. Nodes Chef Server Knife All nodes in the “web” role have “CONVERGED.”
- 84. Sample Role
- 85. Environments • We typically have a PROD and DEV. Maybe QA and STAGING, or others. • Environments are just another label to assign to a node so that it gets the right attributes. • Just like Roles! But with a different name and intent.
- 86. Data Bags • Data Bags are a global source of attributes that any recipe can call upon. • They work great for global attributes • Not so great for secrets like passwords. More on this later.
- 87. Source Control & Chef • Your local chef repo should be cloned from https:// github.com/opscode/chef-repo. • Then commit it to your own Git repo so you can version-control changes to Cookbooks, Roles, Environments, Data Bags, etc. • Now you version-control your infrastructure just like your code itself!
- 88. Attribute Hierarchy • Hard to learn just by looking. Learn by doing.
- 89. Tips & Tricks Also hard to remember. Intended for reference!
- 90. Storing Secrets in Your Infrastructure • This is a very hard problem! Let’s look at some options. • Option 1: The official Chef solution is encrypted data bags. But the main problem is all nodes and the Chef server share the same symmetric encryption key :(. So how do we securely transport and protect that key? • Option 2: Nordstrom uses Chef and created something called Chef Vault to replace the symmetric encryption key of encrypted data bags with public key infrastructure. Works well, but creates the “chicken and egg” problem where a server can’t register itself with chef-vault until it’s bootstrapped, but needs secrets from chef-vault to bootstrap itself. • BEST OPTION for AWS! Option 3: Use Citadel. (https://github.com/ balanced-cookbooks/citadel). Store all your secrets in an S3 bucket. Lock down S3 with AWS IAM Users. Assign each EC2 instance (node) to an IAM Role which automatically grants access to that instance to the S3 buckets we specify. No keys to manage b/c Amazon does it for us!
- 91. Open Source Chef Server Tips • Follow the instructions at http://docs.opscode.com/install_server.html • Go to http://www.getchef.com/chef/install/ to get the URL for the file download • To setup the Fully Qualified Domain Name (i.e. hostname) for Ubuntu, do this: • Setup a DNS name for the server (chef.mybiz.com) • sudo vim /etc/hostname and enter the hostname to handle server reboots • sudo hostname chef.mybiz.com to change the hostname for the current session • Immediately setup a user/pass for yourself so that admin remains a "root" account. • For AWS, a m1.small instance is sufficient for now. • You will need backup and monitoring for this server. • See http://www.getchef.com/blog/2013/03/11/chef-11-server-up-and-running/ for more info. • You can bootstrap Chef Server with Chef Solo! See https://github.com/opscode-cookbooks/ chef-server
- 92. Tips for Setting Up Knife • When you run knife configure --initial use your local paths for the admin.pem and the chef- validator.pem • At some point, you'll need to download files from /etc/chef-server folder on the Chef Server in order to get Knife up and running. • You'll need to modify your knife.rb file (e.g. to point to your cookbooks path) to get things working right. Errors caused by this are not well documented. Here's my knife.rb: Joshs-MacBook-Pro:.chef josh$ vim knife.rb ! ! log_level :info! log_location STDOUT! node_name 'josh'! client_key '/Users/josh/.chef/josh.pem'! validation_client_name 'chef-validator'! validation_key '/repos/chef-repo/.chef/chef-validator.pem'! chef_server_url 'https://chef.projname.mybiz.com'! syntax_check_cache_path '/Users/josh/.chef/syntax_check_cache'! cookbook_path '/repos/chef-repo/cookbooks'! knife[:aws_access_key_id] = "Your AWS Access Key ID"! knife[:aws_secret_access_key] = "Your AWS Secret Access Key"! knife[:region] = "us-west-2"! knife[:vpc_id] = "vpc-XXXXXXX"
- 93. Writing Cookbooks • Having a fast feedback loop is key. Also don’t want to rack up AWS costs. • Ideal environment for writing cookbooks is EC2 tiny instances with a Chef Server or use Chef Solo with Vagrant. • Use test-kitchen to help manage your Vagrant environment. • User berkshelf to help manage cookbook dependencies (if it’s getting out of hand)
- 94. Helpful Ruby Tips • I knew zero Ruby when I started and got by fine. Here are the only confusing things I encountered: • In Ruby, :stringLiteral is called a “symbol” and is equivalent to “stringLiteral”. See http://www.reactive.io/tips/2009/01/11/the-difference- between-ruby-symbols-and-strings/ • A “heredoc” refers simply to a multiline string and is begun by <<EOH and ended with EOH on a newline. • Everything else is pretty straightforward.
- 95. Getting Started & Learning More
- 96. Start with These Milestones 1. Setup Chef Server (hosted or on-premise) 2. Setup Knife 3. Setup Vagrant environment 4. Write (or download) Cookbooks!
- 97. Where to Learn • Start here: https://learnchef.opscode.com. Screencasts are a perfect place to begin. • Then go to http://docs.opscode.com (walkthrough) for more info. • Check out http://gettingstartedwithchef.com as another reference. • Study other people’s cookbooks to get ideas. The postgresql cookbook is very well done. • #chef on IRC was very helpful for me. • The initial learning curve is somewhat steep, but it quickly becomes fun!
- 98. My Favorite Chef Books
- 99. El Fin Now go cook something delicious!