![[Jun AWS 201] Elastic Beanstalk for Startups](https://cdn.slidesharecdn.com/ss_thumbnails/aws201elasticbeanstalkforstartups-130708040156-phpapp02-thumbnail.jpg?width=560&fit=bounds)
DevOps with Elastic Beanstalk - TCCC-2014
- 1. DEVOPS WITH AWS ELASTIC BEANSTALK Scott Colestock Marcato Partners, LLC October 2014
- 2. Elastic Beanstalk: PaaS by AWS • Standardized way to manage deployment of an application • Infrastructure provisioning & scaling • Monitoring / alerting • Application lifecycle (app versions, upgrades, configuration) • You bow out of: • Provisioning raw VMs (IaaS style) • Building up OS with web/app servers and all runtime dependencies • “From-scratch” monitoring, log rolling, health checks, scaling triggers • Competes with OpenShift, Heroku, Azure Cloud Services, Engine Yard, etc.
- 3. All about… Hydrating Infrastructure Quickly • Load balanced farm • Multiple availability zones • Auto scaling with scale-up/ scale-down rules • Per-node health checks • Consolidated log access • Monitoring & alarm conditions • Multiple platforms…
- 4. And Orchestrating the app lifecycle • Start with pre-baked AMI for each platform that has “known good” (cross-tested) set of platform components • Select appropriate EC2 instance type & farm size range per environment • Upload application packages to a version repository from console (or use git-based mechanism or IDE) • Assign versions to application environments to trigger deploy • Rolling upgrades or URL swap to take live • Full set of mechanics for how application should be delivered with correctness, including restarts of app server etc. • Per environment configuration that can be saved/cloned and use by apps at runtime • Application restart available as a consolidated operation across the farm (along with “rebuild environment)
- 5. Conceptual Model • Elastic Beanstalk is classified as a “Deployment & Management” Service in Amazon’s world
- 6. Conceptual Model (2) • Elastic Beanstalk is a region-level service that can house multiple applications • Each application: • Has a name (“SpiffyApp”) • Acts as a container for one or more Environments • (“SpiffyApp-Dev”, “SpiffyApp-Prod”, etc) • Is associated with an archive of Application Versions • Stores “Saved Configurations” of environment settings, which can be reapplied at a later point • Scaling parameters, load balancing parameters, EC2 instance details • Your own key-value pairs that vary by environment (database connection strings, etc.)
- 8. Conceptual Model (4) Gray color indicates environment in transition (restarting or changes being applied) Red color indicates health problem – health check URL failing or instance count below threshold, etc.
- 9. Application Versions • Elastic Beanstalk maintains an archive of all uploaded versions (in S3 bucket) • Version label • Description • Date created • Download link • Where deployed? From this part of the console, you can choose any stored version and trigger a deploy to a particular environment.
- 10. Deploying App Versions – Option 1 • Use the AWS Console What am I uploading? • Depends on the platform • Simple zip for Ruby, WAR for java, Web Deployment Package for .net, etc.
- 11. Deploying App Versions – Option 2 • For node.js / Ruby / Python / PHP, you can choose to use “Eb Command Line Interface” with Git • Google “eb command line download” • After installing, initialize your local git repo git init . • Set path to include location of “eb” tool • “eb init” to initialize AWS keys, region, app name, environment name, and solution stack (one time step) • “eb start” to create do initial app/environment creation (one time step) • After a git commit, “git aws.push” to deploy! • If you change branches & want to change environments, just run “eb init” again
- 12. Deploying App Versions – Option 3 • Use your IDE! Visual Studio & Eclipse have rich plugins that support Elastic Beanstalk
- 13. Deploying App Versions – Option 4 • Use a CI server like Jenkins – using command line tools or plugin support (more on this soon later in the talk…)
- 14. Deployment Mechanics • An application version (package) is delivered to Beanstalk EC2 instances in the way that makes sense for each supported platform • Every web server, app server, and platform has a set of preferred mechanics for getting the package on to each server, executing pre/post scripts, and switching over… • Elastic Beanstalk’s “agent” knows what to do in each case, and reports status back to the Elastic Beanstalk console • In rare cases, you might have to troubleshoot by looking at deployment logs or remotely accessing instances to look at the state in detail
- 15. Application Environments • As many as you need to model your promotion process • + “Disposable” environments for temporary use, etc. • Environments have configuration • Load Balancing & Network • EC2 Details (instance type, availability zones, key pair, etc.) • Your own key-value pairs • Act as scope for log rollup, monitoring, alarms, and events • Are running 1 particular version of your application
- 17. Log Access • At environment level, you can tail all relevant logs (across all servers) or download the whole set (that is currently in rotation) • Beanstalk grabs the right logs for the platform. Can keep you from needing wide access to SSH/RDP • Consider using CloudWatch log aggregation (or a third party like PaperTrail App or Loggly) to do alerting based on log content, & provide richer search/analytics
- 18. Monitoring / Alarms / Alerts • At environment level, you can access a dashboard showing cluster-wide view of: • Average latency, sum of requests, CPU utilization • Network in/out • For any metric, you can also create alarms • CloudWatch would allow you to do this at EC2 instance level – the benefit here is that it applies to all nodes • (Demo)
- 19. Handling Configuration • Good to think about how you want to represent configuration given what Beanstalk provides… • Some platforms and frameworks have you bake knowledge of all possible environments into source-controlled config • At deploy time, you just indicate “Stage” or “Prod” etc. • With Elastic Beanstalk, you may wish to only check in configuration appropriate for local developer desktop, and represent everything else in environment configuration • Beanstalk knows how to present configuration to all platforms it supports in a way that it can be consumed easily
- 20. Configuring .net apps • Recommend: Keep web.config as needed for local development • Then, create config transformation(s) for the build configuration(s) you intend to deploy • Remove key/value pairs that you want configurable in Beanstalk • Example web.release.config snippet: <configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform"> <appSettings> <add key="MongoDBConnectionString" xdt:Transform="Remove" xdt:Locator="Match(key)"/> <add key="MongoDBDatabaseName" xdt:Transform="Remove" xdt:Locator="Match(key)"/> </appSettings> … • Then in Beanstalk configuration UI:
- 21. Application Saved Configurations • Can be created from an Environment Configuration • Can be used as the basis for newly-created environments (create a copy of Dev as a sandbox for a feature branch) • Can be applied to existing environments
- 22. Deploying Apps – Return for Deeper Dive
- 23. Deploying with Jenkins • Contrast: • Deploying with the AWS Console is manual, and requires manual hand-offs • Deploying with eb/git mechanics is great for developer-driven flow, not so great for involving QA or other teams • Deploying with IDE plugins doesn’t guarantee traceability to version controlled code • Deploying with Jenkins and a “build promotion” plugin enables a traceable deployment pipeline where Dev/QA/Product Management can all have visibility/responsibility
- 24. Deploying multiple environments • Most projects run with multiple environments • Local machine • Dev (aka “shared dev”, “integration”) • Test (aka “QA”, “stage”, “UAT”) • Production • We’ve seen that Beanstalk supports multiple environments…
- 25. Deploying multiple environments • Project has to decide to what degree aligning code branches to deployment environments is desirable… • Simple project that deploys frequently and relies heavily on feature branches to vet new code (prior to merge) could use one branch • Use Jenkins to push a given build artifact through environments as confidence grows • More complex project might use feature branches, develop, master, release & hotfix branches ala git flow • Given build artifact may move between environments, but some transitions involve building from an alternate branch
- 26. A light weight branching model approval (Use of promoted builds plugin assumed)
- 28. Github, Jenkins Jobs, & Elastic Beanstalk
- 29. Jenkins to Beanstalk… • Options: • AWS CLI and shell script… • AWSEB Deployment Plugin • CloudBees Amazon Web Services Deploy Engine Plugin • Uses “CloudBees Amazon Web Services Credentials Plugin” • Wiring into job: • You could do this as a post-build step… • But Jenkins “Promoted Builds” plugin adds structure to the process • Promotion criteria • Immediate vs. approval required • When downstream projects build successfully • When specified upstream promotions have completed • Promotions are a “child job” that can be tracked/retried independently
- 30. (Demo) • (Promotion configuration) • (Multiple branches)
- 32. Jenkins Promoted Build Plugin allows you to designate “promotion events” with a symbol – for instance: Unfilled stars = Development Silver stars = Test Gold stars = Production
- 34. A few more things to say regarding Elastic Beanstalk as a whole -
- 35. Direct Access to Servers • It is always possible to get RDP/SSH access to individual servers in a given environment for troubleshooting, etc. • Consider a “sysadmin” security group that opens RDP/TCP to your typical locations, and attach it to an EC2 instance as needed – rather than having these ports open all the time • Do not use this access to alter the configuration of the server. • Script everything through “.ebextensions” or alter the AMI • Don’t configure a (transient!) instance unless you’re experimenting in advance of automating
- 36. Patching • Patching is a mixed story – not as automated as we’d like • You can use .ebextensions file to update on deploy, but its at least possible you will break something: commands: security_updates: command: "yum update -y --security" • You can create new environments (replacing old ones) as Amazon releases new configurations for a given stack
- 37. Generated Names & Artifacts • Beanstalk-generated artifacts within your AWS account often have obscure names • If an ops person doesn’t realize you’re using Elastic Beanstalk, it can slow down troubleshooting • If you attempt to manage Beanstalk-generated artifacts directly, it can cause trouble… • So resist the urge to manage them directly anyway – manage within Beanstalk Console
- 38. Generated Names & Artifacts • Launch Configuration: AMI ID, instance size, security group • Configure in Beanstalk environment creation or instances config • Auto Scaling Group: availability zones, min/max instance counts, scaling thresholds, health check type, scaling history • Configure in Beanstalk environment creation or scaling config • Tags for the ASG do tie you back to Beanstalk environment name • Load Balancers: availability zones, health check, port/listener configuration • Configure in Beanstalk environment creation or scaling/network config • "Instances" tab is fast look at which Beanstalk the ELB is associated with if you need to know.
- 39. .ebextensions configuration • You can use Elastic Beanstalk with a customized AMI, but often better to use deploy-time configuration with .ebextension files • ~/src/SpiffyApp/.ebextensions/ • 01prep.config • 02more.config • What can you do? • Package installation via yum, rubygems, python, rpm, MSI • Additional archive downloads • Creation of files (with permisions) • Creation of users/groups (Linux) • Run shell scripts • Service start/stop (including dependencies) • Even More: • Use resources block to create queues, alarms, elasticache, SNS • (Quite a bit of your infrastructure represented as version controlled code)
- 40. Conclusion • Elastic Beanstalk is a full featured multi-platform PaaS container that can act as a very large infrastructure building block • Full deployment lifecycle • Great start on operations / management • Jenkins + plugins + good branch/environment strategy can allow for a great deployment pipeline • Not just build health – but what build is where? • Getting the right people engaged in promote/deploy mechanics • Not necessarily dev or ops – QA or product management just as easily
Editor's Notes
- #25: I was showing one branch, one environment -
- #26: I was showing one branch, one environment -
- #29: Why have different jobs for different branches? You don’t have to necessarily – but it helps if you are interested in build health, change history, promotion history on a branch-by-branch basis – which you often are. Yes – there is some duplication in these jobs! New plugin that might help with this…