Rise of the machines: Continuous Delivery at SEEK - YOW! Night Summary Slides
4 likes1,772 views
The document outlines Seek's approach to continuous delivery utilizing cloud infrastructure and automation tools, emphasizing practices like immutable servers, blue/green deployments, and infrastructure as code. Key strategies include self-installable applications using native package managers, pre-baked images, and the use of Vagrant for reproducible development environments. Challenges and lessons learned in Windows automation with tools like Packer and DSC are also discussed, highlighting the importance of environment parity for effective continuous delivery.
![[Insert diagram here]](https://image.slidesharecdn.com/riseofthemachinescontinuousdeliveryatseek-summaryslides-150317214541-conversion-gate01/85/Rise-of-the-machines-Continuous-Delivery-at-SEEK-YOW-Night-Summary-Slides-42-320.jpg)
Rise of the machines: Continuous Delivery at SEEK - YOW! Night Summary Slides
- 1. Rise of the Machines Continuous Delivery at SEEK CO PRESENTED BY:
- 2. CD @ SEEK “To the cloud”
- 3. Scorched Earth Deployments ● Create a pristine environment for every release. ● No need to migrate state of environment. ○ ∴ Configuration Management tools not required. ● State should not be changed at runtime. ○ ∴ Immutable Server ● Blue/Green deployment strategy: ○ Monitor health during deployment. ○ Auto-rollback if new environment is unhealthy.
- 4. Automate all the things! ● Delivery team is responsible for the automation of: ○ Server installation ○ Infrastructure creation ○ Deployment ● Automation is impossible without team managed infrastructure.
- 5. Infrastructure as Code ● SEEK use AWS CloudFormation to automate their infrastructure. ● Operations are responsible for the network-level infrastructure, including: ○ VPC ○ Routing ○ Users and Roles ○ Shared services ● Delivery team is responsible for infrastructure within the VPC, such as: ○ Elastic Load Balancer ○ Security Group ○ Auto Scaling Group ○ Launch Configuration ○ Instance Profile
- 6. Server automation ● Provisioning Linux: ○ SSH ⇒ Easy! ● Provisioning Windows: ○ WinRM ⇒ Hard! ○ AWS makes bootstrapping easier: ■ EC2 User Data ■ cfn-init ○ Packer Windows Plugins can be used to create Windows Amazon Machine Images ● Combination of Packer and CloudFormation can be used regardless of target platform
- 7. Packaging ● Applications should be self-installable. ● Application installation should be separate from server automation. ● Installation should be independent of Continuous Deployment tool. ● Native package manager is ideal solution: ○ Linux: apt-get or yum ○ Windows: ■ One-Get (available in next version of Windows) ■ Chocolatey (available now!) ● Package manager is platform independent pattern ○ <package-manager> install <package-name> ● Windows Coding Horror: ○ Interacting with Windows resources is not trivial, custom PowerShell modules or PowerShell DSC Resources make this easier.
- 8. Pre-Baked Images ● Install a package on a base image. ○ But… don’t include configuration. ● Then, create an image (AMI). ● AMI is our environment independent deployable artifact. ● Future is containerised... ○ Docker is the next evolution of our approach.
- 9. Late-binding of context ● Keep your artifact environment agnostic: ○ Frameworks want you to apply configuration at build or package-time. ○ Deploy tools want to apply configuration at install-time. ○ When deploying to the cloud, you should apply configuration at launch-time. ● When AMI is launched it doesn’t know where it is: ○ Need to inject environment-specific context at launch-time. ○ How do you do this?
- 10. Environment Variables A platform independent way of specifying environment-specific config that’s been around since 1979 ● Use environment variables for: ○ DB connection parameters. ○ Location of service endpoints. ● Don’t use them for: ○ Altering runtime behaviour. ● Overriding .NET configuration with environment variables: ○ .NET 4.5.x: write your own Configuration Provider. ○ .NET 4.6: works out of the box. ○ Azure + .NET 4.5.x: works out of the box. ● Service discovery is on the horizon: ○ SEEK is investigating. ○ Externalised config is first step.
- 11. Confidence to Release ● Put enough steps and checkpoints in pipeline to get enough confidence to release. ● Too many checks will result in a slow pipeline and kill the benefits of Continuous Delivery. ● Take things away until you cry. ○ Then... compensate some other way ○ For example, consumer-driven contract testing makes sure we honour API contracts without the need to integrate.
- 12. Ship It! ● Distinction between Deployment and release: ○ Deploy: create Blue environment and deploy application. ○ Release: cut-over from Green to Blue environment. ● Smoke test in Blue environment before release using synthetic transaction. ● Same smoke test can be used to monitor health after release. ● Choose to enable manual release if change set is risky and additional manual testing is required.
- 13. Towards Polyglot Architecture ● Continuous Delivery + Microservices gives teams the ability to choose the right stack for the job. ● For SEEK, CD is opening up the possibility migrating legacy apps from Windows to Linux. ○ Who knows, in the future, running Windows containers will be a viable alternative. ● SEEKs approach to Continuous Delivery provides better logical separation automation steps. ● Time to first commit still a problem and this will only get worse with the proliferation of different stacks and tools.
- 15. Manual Development Environments ■ Are a bad experience for your developers. ■ Are time consuming and therefore costly. ■ Make it difficult for devs to move between projects. ■ Are rarely accurately reconstructed. ■ Encourages manual configuration rather than the more reliable and repeatable practice of Infrastructure-as-code. Can lead to environments that have all of the parts, but not quite in the right order.
- 17. “It works on my box” ■ Sign of environmental divergence. ■ The Butterfly effect: Minor differences in environment configuration can produce obscure bugs in Prod. ■ This issue is compounded when deployments carried out by another person (e.g. Ops).
- 19. Dev vs Production Parity ■ Environment Parity = Confidence. ■ Overcompensate lack of parity with more, possibly manual tests. ■ Environment Parity… ○ enables continuous delivery . ○ requires diligence and automation for accurate replication. ○ encourages infrastructure-as-code.
- 20. Wouldn’t it be great if we... ■ Could use declarative formats for setup automation, to minimize time and cost for new developers joining the project. ■ Had a clean contract with the underlying operating system, offering maximum portability between execution environments.
- 21. Wouldn’t it be great if we... ■ Built apps suitable for deployment on modern cloud platforms, obviating the need for servers and systems admin. ■ Could minimize divergence between development and production, enabling continuous deployment. ■ Could scale up without significant changes to tooling, architecture, or development practices.
- 22. The “12 Factor App” ■ These 5 principles are what loosely describe the 12 factor App, a guideline for modern SaaS products. ■ 12 Factor Apps, Factor #10: Dev/prod parity ■ “Keep development, staging, and production as similar as possible”. ■ Vagrant can help us solve the environment parity issue.
- 23. Vagrant, Docker Compose & Friends Onward to immutable development environments
- 24. Scorched Earth Dev Environments ■ Dev automation typically come in 2 flavours: ○ Use of CM Tools like Puppet, Chef (Boxen, Babushka) ○ Virtual Machine Runners (Vagrant, Docker Compose) ■ CM Tools fraught with challenges: ○ Not isolated ○ Hard to automate across OS/Platforms ○ Doesn’t force developers to automate
- 25. What is Vagrant? “A tool to help create and configure lightweight, reproducible, and portable development environments.” ■ A replacement for manually configured boxes. ○ Enforces infrastructure-as-code. ■ A machine runner (typically a VM or a Container) ■ Separates authoring environment from development runtime ■ Isolated & repeatable
- 26. What is Vagrant? The following is a logical view of how Vagrant interacts with your authoring environment: ■ Vagrant has spun up Virtualbox VM with the name ‘Web’, and is running it on a Mac OSX desktop ■ Code is synchronised from /c/proj/x -> /vagrant of the running environment
- 27. What is Vagrant? ■ The Web application spins up a server using the code in /vagrant, which binds to a private network on the VM on port 80 ■ Port 8080 is exposed to the outside ‘world’ (in this case, a Mac OSX desktop) which forwards requests to port 80 in the running VM ■ The Developer is free to author in whichever IDE of her choosing
- 29. The TAO of Vagrant ■ vagrant up to get a fully working dev runtime. ■ Development occurs in preferred IDE: Visual Studio, Sublime, ViM, with your terminal, your shortcuts - etc. ■ vagrant destroy && vagrant up will destroy the machine and create a new, clean, isolated environment. ■ Workflow repeatable and transferrable to other projects.
- 30. Developing on Ecosystems From monoliths to ecosystems
- 31. From Monolith to Microservices ■ Applications are usually more interesting than a single UI, they are usually connected to things. ○ They form part of a functioning ecosystem. ■ Vagrant good for Monoliths and ecosystems. ■ Explosion of microservices highlights this need.
- 33. From Monolith to Microservices Commentary on Architecture: ■ Multiple components, in disparate technologies ■ Distributed system ■ Layered architecture ■ = complexity Increased complexity increases the number of things that can go wrong. We can combat this with parity and local simulation.
- 34. From Monolith to Microservices ■ Vagrant can replicate ecosystems locally, while still having the benefits of isolation, repeatability, immutability etc. ■ Running ecosystems locally gives us flexibility. ■ Vagrant can simulate or test a number of scenarios, for example to: ○ Model distributed systems behaviour ○ Perform disaster testing; outages, network partitions ... ○ Replace components independently ○ Test OS/DB upgrades ○ Run entire production replica ecosystems, locally
- 35. Local Ecosystem Setup The following is a Logical View of this fictional ecosystem with Web and API applications exposed to the ‘World’ and a database hidden ‘behind the firewall’:
- 37. Local Ecosystem Setup ■ Note that we are running multiple virtualisation technologies here as if they were all running locally ■ Vagrant will spin up 3 local VMs: ○ Web and API applications exposed to the ‘World’ and a database box hidden ‘behind the firewall’: ■ The 4 VMs are able to communicate through a private network on their native ports (80, 80 and 27017 respectively).
- 38. Native to Docker and is fast, but some caveats: ■ Unless on linux, networking is not automatically exposed to the host machine. ■ You need to ensure that your apps are compiled/built within the Docker containers - otherwise you’ll still need to bootstrap your dev box anyway! ■ Hybrid VM/Docker setups not supported. Docker Compose
- 39. Machine Image Pipelines with Packer Where environments are born
- 40. Cattle in Production, Pets elsewhere ■ Dev == Prod - but what about our build/other servers? ■ We don't want cattle in production and pets elsewhere. ■ Quality starts before coding begins ○ Must think about environments we develop, build and deploy to. ■ How do we ensure other environments are configured identically? -> Immutable servers everywhere!
- 41. The Machine Factory ■ How do we go about creating and managing all of these machine images? ■ At SEEK, we needed a way to generate all of these machine images in a sensible, standardised and organised way. ■ Enter: The Machine Factory: ○ “A process for generating machine images for use in different contexts such as development, test, build, prod and so on.” ■ So what does this look like?
- 43. Machine Factory ■ Assembly-line style manufacturing: ○ Input images (AMI, ISO, Docker Image etc.) ○ Apply Build Steps with shared infrastructure code, using CM tools, shell scripts etc. ○ Output Images (AMI, ISO, Docker etc.) ■ Images put in stasis for their destined environment/context where they will be reanimated. ■ CM tools as a code smell: ○ Your application might be too big. For this step, lightweight options usually suffice.
- 44. Tuning the assembling line: Pipelines ■ Booting machines and provisioning from scratch is slow. ■ 2 main strategies: Intermediate images & enrichment. ■ Create intermediate images to shortcut future builds: ○ Install common infra such as log forwarders, statsd, AV, patches/updates etc. ○ Engineering wide (Platform), team wide (Base) or specific Application images.
- 45. Tuning the assembling line: Enrichment ■ Prod should have only the runtime dependencies ○ Better Security. ○ Better Performance. ■ Dev will have the most. This can be expressed as: Prod ⊊ Build Server ⊊ Dev ■ Enrich machines from left to right in your pipeline, feeding previous image into next step.
- 46. Machine Factory @ SEEK Walking the walk
- 47. Applying CD to Talent Search ■ Talentsearch previously part of .NET monolith. ■ Talent Search is a web-based tool that Recruiters use to connect with candidates, and was previously part of the monolith Warner discussed earlier.
- 49. Carving out the relevant bits ■ Isolate scope, cut out vertical slices from the monolith. ■ Enables us to Focus our automation efforts. ■ Broadly this is achieved using 2 approaches: ○ Mocking out downstream APIs for local dev. ○ Consumer Driven Contracts, using Pact. ■ These give us isolation and speed.
- 50. Automating Windows ■ Windows tooling support in past year is much improved. ■ AWS & Vagrant all have first class support. ■ Seek have contributed a number of plugins for Vagrant to make Windows development easier, ■ ...a number of useful chocolatey packages, ■ ...and a large suite of DSC resources to assist configuring Windows machines. ■ And then there is Packer...
- 51. Automating Windows: Packer “A tool for creating identical machine images for multiple platforms from a single source configuration” ■ Packer is the core of our Machine Factory. ○ It will boot a machine from an image such as an ISO ○ Configure it with scripts or CM tools. ○ Export it into multiple formats - such as a Vagrant box, Docker image or an AMI. ■ Uses a simple JSON DSL to easily comprehend and visualise the steps applied in the manufacturing process.
- 52. Automating Windows: Packer ■ Packer uses SSH, which is very fragile on Windows. ■ SEEK have helped create a port of Packer to communicate natively to Windows over WinRM. ■ The result is Packer Community, a set of plugins for Packer that make Windows machine automation a breeze. ■ Using Packer we have automated: ○ our local dev boxes in Vagrant, ○ our build/test/prod servers in AWS, ○ ...across Linux, Windows Server and Mac OSX.
- 53. #winning ■ Dev env < 10 mins for .NET API + UI ■ Dev -> Build -> Prod from Factory ○ Vagrant: Vbox, Parallels, Mac + Windows ○ AWS ■ Commit to Release in 7 mins. ■ Proven ‘vertical slices’ strategy.
- 54. Bonus: Challenges & Lessons on Windows Tips on effective use of Vagrant, Packer and DSC.
- 55. Vagrant ■ Other authoring environment dependencies = anti-pattern. ○ Installing VM provider (e.g. Virtualbox) + Vagrant should be the goal ○ Consider scripting the install (DSC, BoxStarter, Powershell scripts etc.) if you can’t avoid. ■ Shared folders and IIS Applications don’t play nicely ○ Use rsync synced folder type instead ■ When provisioners get too big/slow, move into a Packer script - vagrant up should be fast and reliable
- 56. Vagrant ■ Always set $VAGRANT_HOME on multi-partitioned machines due to disk space restrictions (n+1 space requirements) ■ Distributing Images ○ Use Versions + Atlas, if possible ○ S3 also good option ■ GUI Testing (e.g. Protractor tests) ○ Consider having a GUI fallback (at the cost of slightly bigger images) ○ X-Forwarding on *nix hosts not ideal
- 57. Vagrant ■ Cross-OS concerns do occasionally come into play: ○ NPM for example has issues writing to a linux networked file system where the underlying host is Windows. ■ Domain access can be problematic and increases complexity - avoid if possible. ○ Use Vagrant Windows Domain plugin if you can’t. ■ Use multiple NICs with private_network if application requires multiple SSL certificates.
- 58. Vagrant Box cleanup: ■ Clean up after yourself and remove any binaries/installers/cruft not required for day-to-day use ■ Defrag and compress virtual disk images if possible ■ Matt Wrock has a number of good tips on how to cut it right down.
- 59. Vagrant Build .NET apps in Vagrant without Visual Studio: ■ Install MS Build Tools and VS Agents ■ Package reference assemblies from VS installation into guest machine. Install the remote debugger and port forward on 4018 to debug from Visual Studio
- 60. DSC ■ Expect pain/effort, it’s in its infancy and is immature. ■ Error messages can be very unhelpful/confusing, and you should be prepared to get comfortable with Windows debugging in the event viewer and the CLI. ■ IMHO, DSC is still a better choice than the *nix open source alternatives such as Chef/Ansible (although support is improving).
- 61. DSC ■ Use the Vagrant and the DSC Plugin to iterate on a box; Vagrant allows you to test your DSC configurations in a clean environment and test any potential issues with conditions/ordering, without polluting your host environment ■ Prefer the feature rich SEEK-Jobs/DSC community resources over the Powershell team (experimental) xModules ■ Avoid the Script resource - It is rarely a good idea to use it. Write your own custom resource instead.
- 62. Packer ■ Don't try to do it all in one go. Build complexity ■ When building a new image, create scripts, run them manually and then automate once you’re comfortable they’ re working as expected. ■ Iteration times can be slow, particularly when booting and installing an OS ○ Create intermediate boxes to build off to avoid that overhead. ■ Using Vagrant to test those scripts is often a good starting point.
- 63. Packer ■ Sysprep Windows machines so licenses don’t expire. ■ If you’re creating a Windows image, Use Packer Community (only if you can’t use Linux! :)). ■ Don’t forget to disable Windows Updates (at least for dev machines). ■ Windows boxes often need restarting, use the restart- windows Packer Community provisioner to do so safely.
- 64. Packer ■ Build Packer images in pipelines - feed output of Packer runs into the start of another Packer run. ■ For example, use the Virtualbox ISO builder to boot and install a Windows machine with updates applied. ■ That box can now be fed into the Virtualbox OVF builder which runs the application specific provisioners (typically this is the more error prone step).
- 65. Thank you @SEEK_Geek CO PRESENTED BY: @terrencebenade @dius_au @warnergodfrey @matthewfellows
- 66. Resources ■ Vagrant (http://vagrantup.com) ■ The TAO of Vagrant (http://mitchellh.com/the-tao-of-vagrant) ■ Vagrant DSC Plugin (https://github.com/mefellows/vagrant-dsc/) ■ Vagrant Windows Domain Plugin (https://github.com/SEEK- Jobs/vagrant-windows-domain) ■ DSC (http://blogs.technet. com/b/privatecloud/archive/2013/08/30/introducing-powershell- desired-state-configuration-dsc.aspx) ■ Read the free DSC eBook (https://onedrive.live.com/? cid=7f868aa697b937fe&id=7F868AA697B937FE!107) for a small but comprehensive guide to DSC
- 67. Resources ■ SEEK-Jobs/DSC resource (https://github.com/SEEK-Jobs/DSC/) ■ Setting up a server for use with DSC (see packer templates https: //github.com/mefellows/packer-community-templates) ■ Packer (http://packer.io) ■ Packer Windows Plugins (aka Packer for Windows https://github. com/packer-community/packer-windows-plugins) ■ Sample Windows Packer Templates (https://github. com/mefellows/packer-community-templates) ■ Legacy Windows Packer Templates and provisioning scripts (Uses OpenSSH, however there is plenty of inspiration to be found - https://github.com/joefitzgerald/packer-windows)