stackconf 2022: Infrastructure Automation (anti) patterns
0 likes51 views
Kris Buytaert presents at Stackconf 2022, discussing the evolution of infrastructure as code (IaC) and its importance in DevOps, highlighting patterns and tools used over the past 20 years. He emphasizes the necessity of using IaC for reproducible environments and preventing configuration drift, while addressing common patterns and pitfalls in automation. The talk advocates for treating configuration automation as code, aligning with best development practices to improve software delivery quality.
stackconf 2022: Infrastructure Automation (anti) patterns
- 1. Kris Buytaert @krisbuytaert Infrastructure as Code Patterns StackConf 2022, Berlin
- 2. • I used to be a developer • Then I became an Ops person • Chief Trolling/Travel/Technical Officer @ Inuits.eu • Chief Yak Shaver @ o11y.eu • Organiser of #devopsdays, #cfgmgmtcamp, #loadays, ... • Everything is a Freaking DNS Problem • Cofounder of all of the above • DNS : devops needs sushi • @krisbuytaert on twitter/github Who am I ? @krisbuytaert
- 4. A global movement to improve the quality of software delivery leveraging Open Source experiences , started in Gent in 2009 Devops definition @krisbuytaert
- 5. We love running in circles @krisbuytaert
- 6. • 1996 : Manual installation • 2001 : Mondo Rescue (reproducable single instance) • 2003 : System Imager , fast reproducable "identical" Infrastructure • Image Sprawl • "Overrides" • 2005 : JeOS + Early IAC (CfEngine / Kickstart / FAI) • 2010 : Desired State infrastructure as code , Puppet, Chef etc • 2015 : Cloud provisioning , Terraform etc .. Deploying an Infrastructure @krisbuytaert
- 7. • 1 (Snowflake) server • Multiple (snowflake) servers (cluster nodes) • Virtualization • Cloud • Containers 20+ years of changes @krisbuytaert
- 8. • Manual error • Differences in nodes due to manual changes • Not reproducable • Humans • Scale Some of The problems @krisbuytaert
- 10. • Manually Crafted Snowflake • Copied and modified • Copied and modified • Impossible to trace origin • Impossible to reproduce • Patching / Security nightmare Image Sprawling & Golden Images vs JeOS @krisbuytaert
- 11. FROM ubuntu RUN apt-get update RUN apt-get install -y git RUN git clone [email protected]:prometheus/prometheus.git RUN make install What's wrong with this GW-Basic code ? @krisbuytaert
- 12. Infrastructure as What ? @krisbuytaert
- 13. • Treat configuration automation as Code • Model your infrastructure • Development best practices • Model your infrastructure • Pipeline your Code • Test your Code Infrastructure as Code @krisbuytaert
- 14. • Reproducable Environments • You never deploy 1x • Local Test environments • Development / User Acceptance / Testing / Shadow / Production / DR • HA <- Keep cluster configuration in sync • Prevent Configuration drift • Cheap Disaster recovery • ... Infrastructure as Code Goals @krisbuytaert
- 15. 3 types of tools needed • Provisioning • Desired state • Orchestration 2022 IaC Requirements @krisbuytaert
- 16. • Create me an instance of application X • Container instance • VM instance • Service X configuration via API e.g Terraform , Pulumi Provisioning @krisbuytaert
- 17. • Ensure that this file present / service is running • With these permissions • Always / Verified e.g Chef, Puppet, Salt Desired state @krisbuytaert
- 18. • Non frequent • Trigger action X on resource Y based on characteristics A,B and or C • First do X here then do Y there • One off actions e.g Ansible, Salt, Puppet Bolt Orchestration @krisbuytaert
- 19. • In computing, an idempotent operation is one that has no additional effect if it is called more than once with the same input parameters. • Not idempotent : echo “root=disabled” >> /etc/someconfigfile Idempodency @krisbuytaert
- 20. Frequent enforcing desired state with reporting = Automated population of a CMDB (Single source of truth) , with up to date data Side Effect : Having a single source from where to generate dynamic configuration eg. reverse proxy , firewall rules, monitoring configuration, backup configuration IaC & CMDB @krisbuytaert
- 21. • Do you want a reproducible infrastructure ? • Do you want to reduce manual error ? • Do you want to achieve desired state ? • Do you want to have automated configuration for monitoring, metrics, backups, security etc ? • Do you want to trivially spin up different identical environments (dev/uat/prod/dr) • Do you want to be able to detect and prevent config drift ? • Do you want to prevent image sprawl ? Questions to ask : @krisbuytaert
- 23. • Code is not in source control • Bolt / Ansible / Terraform is triggered from the admin's local machine • Works for individual admin • Doesn't Scale • Prevents Collaboration Pattern: It works from my homedir @krisbuytaert
- 24. Improvement over $prev, but still doesn't Scale Collaboration has been improved. Pattern: It works from my workstation but its in git @krisbuytaert
- 25. • Just the operating System • Maybe some basic services • Application is untouched • Partial source of maybes • Not suitable for derived configuration • Kill your Silos Pattern: We only automate the OS @krisbuytaert
- 26. • Operating System • Middle Ware • Application Pattern: We automate all the things @krisbuytaert
- 27. • Package • Config • Service Pattern: PCS @krisbuytaert
- 28. • PCS • Non Functional Requirements • Monitoring • Backups • External dependencies • Metrics • ... Pattern: PCS NFR @krisbuytaert
- 29. • Package • Config Files • User Generated Data Pattern: PCS UGC @krisbuytaert
- 30. Pattern: Reproducable and Scalable Infrastructure @krisbuytaert
- 31. • Ops use Chef but Devs use Java templating tool or • Terraform but Manual changes from UI. Variants: • Non centralized code base, people run code from local forks undoing changes made by colleagues • Templating tool is not idempotent (ruby sort ? / XML) • Bonus points if the tool restarts a service on a change Pattern: Multiple tools overwrite changes @krisbuytaert
- 32. • Code is checked into a git repository • The only way to apply is by central user Pattern: Centralised apply , from source control @krisbuytaert
- 33. • Tool is run once at deploy time then yolo • manual changes afterwards are not identified or reversed • State is not enforced, people are afraid to rerun playbooks • Now we don't dare to update anymore Orchestraition only mode, no desired state AKA Yolo or Integrator mode Pattern: We haven't run it for 6 months @krisbuytaert
- 34. • Reporting says it's broken, we are not going to fix it • Its been disabled for weeks, local modifications break it • See Config Drift Pattern: It's broken anyhow . @krisbuytaert
- 35. • X < 120 • Periodically we run ensure the state of our Infrastructure • We prevent config Drift • We actively monitor broken runs • We actively monitor change frequency We have achieved desired state of our infra. Pattern: We run it every X minutes @krisbuytaert
- 36. • We setup everything, it worked for a while • Then we broke service A , • Then something unrelated broke service B • Automation got blamed • We are not allowed to enable it again • Team X doesn't trust automation, • Often Team X = Security • We manually make the changes now , but we have visibility on drift. • We have semi desired state Pattern: We run it in nooop now @krisbuytaert
- 37. Pattern: We have a pipeline @krisbuytaert
- 38. What is an Environment , a stack, a platform ? A logically split of set of servers that together offer a service, Typically existing in dev/uat/prod flavours Environments @krisbuytaert
- 39. • Limiting Blast radius • Easy to spin up multiple variants • Feature flagged code introduction • And promotion of features • Clear line of ownership (team bound) • Same code , different config per environment • Easier release management Pattern : Using Environments @krisbuytaert
- 40. • Huge blast radius • Potential Scaling issues • Difficult to "promote" changes • Upgrade everthing or write host based if then case spaghetti • Code duplication : role_app_dev, role_app_uat, role_app_prod Pattern : Not using environments @krisbuytaert
- 41. • Typically with no environments • Environment specific config is hardcoded • See Config drift Pattern: Config is hardcoded in the code @krisbuytaert
- 42. • The same codebase is used • Only the config data differs • Multiple Environments possible Pattern: Config Data defines config per environment @krisbuytaert
- 43. • Code is run • We will see the errors when they happen • Code is not tested Pattern: It we don't test @krisbuytaert
- 44. • Every config run • Collect information of the run, • Store that information • Be able to query a central source with live, correctly, machine generated data • Reconfigure services based on changes in that data • Full lifecycle • Decomission instance => Automatically Clean out the data Pattern: Single source of truth @krisbuytaert
- 45. SSOT : e.g Puppet Exported Resources @krisbuytaert
- 46. • Our Ansible playbook does a git commit to a config repo • That repo gets checked out on the monitoring server • Humans also modify this config • Information is scattered over different tools • you can't leverage the real power of automation , no dynamic reconfiguration of resources therefore configuration is often done manually , but through a tool Pattern: No single source of truth @krisbuytaert
- 47. We have one repo with all • code • passwords • software binaries Pattern: One repo for all the things @krisbuytaert
- 48. "Blue printing" Building a UI on top of a provisioning framework is typically an intermediate step before people realize they either need to build in all AAA functionality and buy a bloated tool. Or realize git + Terraform solves these problems in a better way. Pattern: Provisioning from a UI @krisbuytaert
- 49. • We have written all code in house ? • What do you mean there are modules on github to manage apache ? • We spent about 3 months building our own Pattern: we ignore the existing codebase group @krisbuytaert
- 50. • Aka We do NOT understand Continuous Integration • We have mapped our environments to git branches • Obviously they are long running • We are cherry picking changes from uat to prod • We don't test those combinations • We are in merge hell now Puppet r10k is an antipattern. So is running your gitops from multiple branches. Pattern: We love Branching @krisbuytaert
- 51. Puppet & Chef Terraform & Pulumi Pattern: We want a full language @krisbuytaert
- 52. • All of these people say they are doing infrastructure as Code • Some with 0 real benefit • Find the patterns you like / dislike • Implement or remove the ones you please • My preferences might not fit your use case Lessons Learned @krisbuytaert
- 53. Kris Buytaert @krisbuytaert [email protected] Essensteenweg 31 2930 Brasschaat Belgium Contact: [email protected] +32-3-8082105