SlideShare a Scribd company logo
Implementing DevOps Automation
Best Practices and Common Mistakes
Given by Derek C. Ashmore
Agile+DevOps East 2023
November 8, 2023
©2023 Derek C. Ashmore, All Rights Reserved 1
Who am I?
• Professional Geek
since 1987
• AWS since 2010
• Azure since 2017
• Specialties
• Application
Transformation
• Infrastructure
Automation
• Yes – I still code!
©2023 Derek C. Ashmore, All Rights Reserved 2
Discussion Resources
• This slide deck
– https://www.slideshare.net/derekashmore/presentations
• Slide deck has hyper-links!
– Don’t bother writing down URLs
• I take questions
– For those online, contact me on LinkedIn
©2023 Derek C. Ashmore, All Rights Reserved 3
Agenda
Intro and
Level Set
DevOps Best
Practices
DevOps
Management
Summary /
Q&A
©2023 Derek C. Ashmore, All Rights Reserved 4
DevOps Managed Infrastructure
• 99+% Infrastructure as Code (IaC)
• Manual changes
– Increase errors
– Increase unwanted differences between
environments
– Increase admin workload
• Scripted/Coded changes
– Larger upfront cost, but…..
– Less busywork
– Leverage Others Work
– Decreases Errors
– Errors fixed in one place
– Eliminates unwanted differences
– Change history (with source control)
©2023 Derek C. Ashmore, All Rights Reserved 5
Tales from the Field
• Large Consumer Product Firm
– Rebuilds entire cloud footprint every
two weeks
• Large Fast-food Franchise
– Easy to add new business unit spokes
– Security / Guardrails built in
– Internet ingress/egress
– On premises network connectivity
• Large Financial Institution
– Mobile App Cloud and application
footprint
• blue/green capability
©2023 Derek C. Ashmore, All Rights Reserved
DevOps Automation Categories
➢Network / non-application specific infrastructure
▪ Virtual Networks/VPCs and subnets
▪ Route tables, Network peering
▪ Security groups / NSGs
➢Application infrastructure
▪ Relational databases
▪ Serverless constructs
➢Security privileges and policies
▪ IAM Roles and privilege grants
➢Virtual machine image production
▪ Produce machine images for teams to use
▪ Docker image production is similar conceptually
©2023 Derek C. Ashmore, All Rights Reserved 7
Agenda
Intro and
Level Set
DevOps Best
Practices
DevOps
Management
Summary /
Q&A
©2023 Derek C. Ashmore, All Rights Reserved 8
Discipline is Key
➢Discipline required differs per maturity level
➢Source Code Management
➢Source Code Structure
➢Deployment Management (CI/CD Pipelines)
➢Avoid Manual Changes
➢Testing Strategy
©2023 Derek C. Ashmore, All Rights Reserved 9
Automation Usage Evolution
• In the beginning
– Use Source Control
• As #Coders grow
– Feature branches
– CI/CD Pipelines
• As #Configurations grow
– Separate repo for modules
– Implement versioning
• Never use main/master!
• Further reading
©2023 Derek C. Ashmore, All Rights Reserved
Feature Branching
• DevOps Team Discipline is Key
• Feature Branches
– Never edit main/master directly!
– Update using Pull Requests
• Should live less than one day!
– Single targeted enhancement
– One developer only
– Long-lived branches prone to
merge conflicts
– Squash commits on merge
• Further reading
©2023 Derek C. Ashmore, All Rights Reserved
Code in Reusable Modules
➢Advantages are
▪ Small blast radius
▪ More easily tested
▪ Economies of scale
➢Example reusable modules
▪ Kubernetes Cluster
▪ Virtual Machine
▪ Virtual Networks and Subnets
▪ S3/Storage accounts
▪ Serverless services/functions
➢100+ Modules in all
▪ Used in 400+ pipelines
➢Tested in merge to master
©2023 Derek C. Ashmore, All Rights Reserved 12
CI/CD Pipelines
• Provides consistent runtime
environment
– Software version
– Cloud security policy
• Audit history / Admin security
• Pipeline approvals
– Force Plan/Dry-Run execution
– Force manual approval before
changing the environment
©2023 Derek C. Ashmore, All Rights Reserved
Manual Intervention Requirements
➢Some companies require manual intervention
▪ Often dictated by company policy
▪ Examples include
▪ Requiring DNS entries to be manually entered
▪ Separate group allocates security privileges
▪ On-premises connectivity
➢IaC depending on manual intervention cannot have automated tests
▪ Localize the manual intervention requirements
©2023 Derek C. Ashmore, All Rights Reserved 14
Accommodating Manual Processes
©2023 Derek C. Ashmore, All Rights Reserved 15
Infrastructure Code Testing
➢IaC is code!
▪ Housed in source control
▪ Often changed and released
▪ Needs testing like any other code
➢IaC change can have negative impact
▪ Environment outages
▪ End-user internet connectivity outage
▪ Application outages
▪ Testing team delayed for four days
➢Testing IaC can minimize negative impact
©2023 Derek C. Ashmore, All Rights Reserved 16
Infrastructure Code Testing Differences
➢IaC != Application Code
▪ IaC requires external resources (e.g. Cloud) to run
▪ In-process unit testing often not possible
➢Limited localized (in-process) testing
▪ Generally limited to syntax checks
▪ Terraform validation
▪ Ansible Dry Runs
▪ IDE syntax checks
➢Most testing is “integration” testing
©2023 Derek C. Ashmore, All Rights Reserved 17
Infrastructure Code Testing Challenges
➢Friction
▪ Harder to write/maintain
➢Dependencies
▪ Managed by other teams
➢Testing costs
▪ Use Sandbox – tear-down after tests
➢Manual intervention requirements
▪ Not possible to automate tests
©2023 Derek C. Ashmore, All Rights Reserved 18
Lack of Discipline Causes…
➢Unplanned Work
▪ Change due to automatic upgrades
▪ Unintended consequences
➢Accidental over-writing changes of others
▪ Merge conflicts
▪ Changes deployed from unmerged branches
➢Increased defect rate
➢Configuration Drift caused by manual change
©2023 Derek C. Ashmore, All Rights Reserved 19
Frequent Mistakes in the Field…
➢Lack of testing for common IaC code
▪ Testing for one use of common code is not sufficient!
▪ Often breaks other consumers of the common code
➢Creating a blast radius for IaC that’s too large
▪ Can’t make targeted changes without unintended consequences
➢Treating common IaC code as an enforcement mechanism
▪ Decouple policy enforcement and naming conventions
▪ Common IaC is a productivity enhancer only
©2023 Derek C. Ashmore, All Rights Reserved 20
Agenda
Intro and
Level Set
DevOps Best
Practices
DevOps
Management
Summary /
Q&A
©2023 Derek C. Ashmore, All Rights Reserved 21
Management is different too!
• Instead of
– Manual reviews/approvals
• Automate guardrails
• Automate testing
• Whitelist cloud services
• Consider continuous
delivery/deployment
– Capacity planning up front
• Monitor cost increases and
investigate
– Mandating policy changes by
edict
• Automate policy enforcement
©2023 Derek C. Ashmore, All Rights Reserved
Things that Don’t work
• Adding Manual
Approvals/Reviews
– Kills velocity and productivity
– Stops innovation
– Creates bottlenecks
• Forcing manual procedures
– Attempt to “expedite”
– Creates technical debt
©2023 Derek C. Ashmore, All Rights Reserved
Things that Work!
• Declare War on manual approvals
– Favor automated guardrails
– Automate oversight
• Decentralize Cloud Management
– Let app teams manage app infrastructure
• Don’t be a bottleneck
– Leave App teams to innovate
• Create a Service Catalog
– Automate whitelisting of services
– Create reasonable process for new services
• Legal reviews (HIPPA, GDPR, etc.)
• Fund automation
– You won’t make it manually!
• DevOps Team Discipline
– Automation needs SDLC just like applications
– Source management is key
©2023 Derek C. Ashmore, All Rights Reserved
Thank you!
• Derek Ashmore:
– Blog: www.derekashmore.com
– LinkedIn: www.linkedin.com/in/derekashmore
• Connect Invites from attendees welcome
– Twitter: https://twitter.com/Derek_Ashmore
– GitHub: https://github.com/Derek-Ashmore
– Book: http://dvtpress.com/
• Please fill out the evaluation form!
©2023 Derek C. Ashmore, All Rights Reserved 25

More Related Content

Similar to Implementing DevOps Automation Best Practices and Common Mistakes (20)

PPTX
Ice breaker with dev ops
Mukta Aphale
 
PDF
DevOps Roadtrip NYC
VictorOps
 
PPT
icebreakerwithdevops-150218112943-conversion-gate02
Manohar Kumar
 
PDF
DevSecOps: A Secure SDLC in the Age of DevOps and Hyper-Automation
Alex Senkevitch
 
PDF
DevOps and DevSecOps, Incident Management
ShriniKulkarni
 
PDF
How DevOps Development Companies Streamline Operations.pdf
Agile Infoways LLC
 
PPTX
Are your DevOps and Security teams friends or foes?
Reuven Harrison
 
PDF
Scale security for a dollar or less
Mohammed A. Imran
 
PPTX
DevOps in Network Engineering ( Network Automation using DevNet)
akintunero101
 
PPTX
Implementing DevOps Automation in Your Organization.pptx
HashStudiozTechnolog1
 
PDF
Devops Interview Question PDF By ScholarHat
Scholarhat
 
DOCX
A Comprehensive Guide with DevOps Infrastructure Management Services at HEX64...
HEX64
 
PPTX
DevOps Practices: Bridging the Gap Between Development and Operations
AyeshaSharif19
 
PDF
Dear security, compliance, and auditing: We’re sorry. Love, DevOps | DevNatio...
Red Hat Developers
 
PPTX
SecDevOps: The New Black of IT
CloudPassage
 
PPTX
What is dev ops?
Mukta Aphale
 
PPTX
DevOps Training | DevOps Online Training in Hyderabad.pptx
ranjithvisualpath44
 
PPTX
DevSecOps Best Practices-Safeguarding Your Digital Landscape
stevecooper930744
 
PDF
Leveraging DevOps for Faster and Scalable Deployments - Keynote.pdf
PaschalOruche1
 
PDF
Confoo-Montreal-2016: Controlling Your Environments using Infrastructure as Code
Steve Mercier
 
Ice breaker with dev ops
Mukta Aphale
 
DevOps Roadtrip NYC
VictorOps
 
icebreakerwithdevops-150218112943-conversion-gate02
Manohar Kumar
 
DevSecOps: A Secure SDLC in the Age of DevOps and Hyper-Automation
Alex Senkevitch
 
DevOps and DevSecOps, Incident Management
ShriniKulkarni
 
How DevOps Development Companies Streamline Operations.pdf
Agile Infoways LLC
 
Are your DevOps and Security teams friends or foes?
Reuven Harrison
 
Scale security for a dollar or less
Mohammed A. Imran
 
DevOps in Network Engineering ( Network Automation using DevNet)
akintunero101
 
Implementing DevOps Automation in Your Organization.pptx
HashStudiozTechnolog1
 
Devops Interview Question PDF By ScholarHat
Scholarhat
 
A Comprehensive Guide with DevOps Infrastructure Management Services at HEX64...
HEX64
 
DevOps Practices: Bridging the Gap Between Development and Operations
AyeshaSharif19
 
Dear security, compliance, and auditing: We’re sorry. Love, DevOps | DevNatio...
Red Hat Developers
 
SecDevOps: The New Black of IT
CloudPassage
 
What is dev ops?
Mukta Aphale
 
DevOps Training | DevOps Online Training in Hyderabad.pptx
ranjithvisualpath44
 
DevSecOps Best Practices-Safeguarding Your Digital Landscape
stevecooper930744
 
Leveraging DevOps for Faster and Scalable Deployments - Keynote.pdf
PaschalOruche1
 
Confoo-Montreal-2016: Controlling Your Environments using Infrastructure as Code
Steve Mercier
 

Recently uploaded (20)

PPTX
MiniTool Power Data Recovery Full Crack Latest 2025
muhammadgurbazkhan
 
PDF
HiHelloHR – Simplify HR Operations for Modern Workplaces
HiHelloHR
 
PPTX
Feb 2021 Cohesity first pitch presentation.pptx
enginsayin1
 
PPTX
An Introduction to ZAP by Checkmarx - Official Version
Simon Bennetts
 
PPTX
Comprehensive Guide: Shoviv Exchange to Office 365 Migration Tool 2025
Shoviv Software
 
PDF
Why Businesses Are Switching to Open Source Alternatives to Crystal Reports.pdf
Varsha Nayak
 
PDF
유니티에서 Burst Compiler+ThreadedJobs+SIMD 적용사례
Seongdae Kim
 
PDF
Streamline Contractor Lifecycle- TECH EHS Solution
TECH EHS Solution
 
PDF
Unlock Efficiency with Insurance Policy Administration Systems
Insurance Tech Services
 
PPTX
Engineering the Java Web Application (MVC)
abhishekoza1981
 
PDF
Efficient, Automated Claims Processing Software for Insurers
Insurance Tech Services
 
PPTX
Migrating Millions of Users with Debezium, Apache Kafka, and an Acyclic Synch...
MD Sayem Ahmed
 
PDF
Beyond Binaries: Understanding Diversity and Allyship in a Global Workplace -...
Imma Valls Bernaus
 
PPTX
Human Resources Information System (HRIS)
Amity University, Patna
 
PPTX
Fundamentals_of_Microservices_Architecture.pptx
MuhammadUzair504018
 
PDF
Linux Certificate of Completion - LabEx Certificate
VICTOR MAESTRE RAMIREZ
 
PDF
Salesforce CRM Services.VALiNTRY360
VALiNTRY360
 
PPT
MergeSortfbsjbjsfk sdfik k
RafishaikIT02044
 
PDF
Automate Cybersecurity Tasks with Python
VICTOR MAESTRE RAMIREZ
 
PPTX
3uTools Full Crack Free Version Download [Latest] 2025
muhammadgurbazkhan
 
MiniTool Power Data Recovery Full Crack Latest 2025
muhammadgurbazkhan
 
HiHelloHR – Simplify HR Operations for Modern Workplaces
HiHelloHR
 
Feb 2021 Cohesity first pitch presentation.pptx
enginsayin1
 
An Introduction to ZAP by Checkmarx - Official Version
Simon Bennetts
 
Comprehensive Guide: Shoviv Exchange to Office 365 Migration Tool 2025
Shoviv Software
 
Why Businesses Are Switching to Open Source Alternatives to Crystal Reports.pdf
Varsha Nayak
 
유니티에서 Burst Compiler+ThreadedJobs+SIMD 적용사례
Seongdae Kim
 
Streamline Contractor Lifecycle- TECH EHS Solution
TECH EHS Solution
 
Unlock Efficiency with Insurance Policy Administration Systems
Insurance Tech Services
 
Engineering the Java Web Application (MVC)
abhishekoza1981
 
Efficient, Automated Claims Processing Software for Insurers
Insurance Tech Services
 
Migrating Millions of Users with Debezium, Apache Kafka, and an Acyclic Synch...
MD Sayem Ahmed
 
Beyond Binaries: Understanding Diversity and Allyship in a Global Workplace -...
Imma Valls Bernaus
 
Human Resources Information System (HRIS)
Amity University, Patna
 
Fundamentals_of_Microservices_Architecture.pptx
MuhammadUzair504018
 
Linux Certificate of Completion - LabEx Certificate
VICTOR MAESTRE RAMIREZ
 
Salesforce CRM Services.VALiNTRY360
VALiNTRY360
 
MergeSortfbsjbjsfk sdfik k
RafishaikIT02044
 
Automate Cybersecurity Tasks with Python
VICTOR MAESTRE RAMIREZ
 
3uTools Full Crack Free Version Download [Latest] 2025
muhammadgurbazkhan
 
Ad

Implementing DevOps Automation Best Practices and Common Mistakes

  • 1. Implementing DevOps Automation Best Practices and Common Mistakes Given by Derek C. Ashmore Agile+DevOps East 2023 November 8, 2023 ©2023 Derek C. Ashmore, All Rights Reserved 1
  • 2. Who am I? • Professional Geek since 1987 • AWS since 2010 • Azure since 2017 • Specialties • Application Transformation • Infrastructure Automation • Yes – I still code! ©2023 Derek C. Ashmore, All Rights Reserved 2
  • 3. Discussion Resources • This slide deck – https://www.slideshare.net/derekashmore/presentations • Slide deck has hyper-links! – Don’t bother writing down URLs • I take questions – For those online, contact me on LinkedIn ©2023 Derek C. Ashmore, All Rights Reserved 3
  • 4. Agenda Intro and Level Set DevOps Best Practices DevOps Management Summary / Q&A ©2023 Derek C. Ashmore, All Rights Reserved 4
  • 5. DevOps Managed Infrastructure • 99+% Infrastructure as Code (IaC) • Manual changes – Increase errors – Increase unwanted differences between environments – Increase admin workload • Scripted/Coded changes – Larger upfront cost, but….. – Less busywork – Leverage Others Work – Decreases Errors – Errors fixed in one place – Eliminates unwanted differences – Change history (with source control) ©2023 Derek C. Ashmore, All Rights Reserved 5
  • 6. Tales from the Field • Large Consumer Product Firm – Rebuilds entire cloud footprint every two weeks • Large Fast-food Franchise – Easy to add new business unit spokes – Security / Guardrails built in – Internet ingress/egress – On premises network connectivity • Large Financial Institution – Mobile App Cloud and application footprint • blue/green capability ©2023 Derek C. Ashmore, All Rights Reserved
  • 7. DevOps Automation Categories ➢Network / non-application specific infrastructure ▪ Virtual Networks/VPCs and subnets ▪ Route tables, Network peering ▪ Security groups / NSGs ➢Application infrastructure ▪ Relational databases ▪ Serverless constructs ➢Security privileges and policies ▪ IAM Roles and privilege grants ➢Virtual machine image production ▪ Produce machine images for teams to use ▪ Docker image production is similar conceptually ©2023 Derek C. Ashmore, All Rights Reserved 7
  • 8. Agenda Intro and Level Set DevOps Best Practices DevOps Management Summary / Q&A ©2023 Derek C. Ashmore, All Rights Reserved 8
  • 9. Discipline is Key ➢Discipline required differs per maturity level ➢Source Code Management ➢Source Code Structure ➢Deployment Management (CI/CD Pipelines) ➢Avoid Manual Changes ➢Testing Strategy ©2023 Derek C. Ashmore, All Rights Reserved 9
  • 10. Automation Usage Evolution • In the beginning – Use Source Control • As #Coders grow – Feature branches – CI/CD Pipelines • As #Configurations grow – Separate repo for modules – Implement versioning • Never use main/master! • Further reading ©2023 Derek C. Ashmore, All Rights Reserved
  • 11. Feature Branching • DevOps Team Discipline is Key • Feature Branches – Never edit main/master directly! – Update using Pull Requests • Should live less than one day! – Single targeted enhancement – One developer only – Long-lived branches prone to merge conflicts – Squash commits on merge • Further reading ©2023 Derek C. Ashmore, All Rights Reserved
  • 12. Code in Reusable Modules ➢Advantages are ▪ Small blast radius ▪ More easily tested ▪ Economies of scale ➢Example reusable modules ▪ Kubernetes Cluster ▪ Virtual Machine ▪ Virtual Networks and Subnets ▪ S3/Storage accounts ▪ Serverless services/functions ➢100+ Modules in all ▪ Used in 400+ pipelines ➢Tested in merge to master ©2023 Derek C. Ashmore, All Rights Reserved 12
  • 13. CI/CD Pipelines • Provides consistent runtime environment – Software version – Cloud security policy • Audit history / Admin security • Pipeline approvals – Force Plan/Dry-Run execution – Force manual approval before changing the environment ©2023 Derek C. Ashmore, All Rights Reserved
  • 14. Manual Intervention Requirements ➢Some companies require manual intervention ▪ Often dictated by company policy ▪ Examples include ▪ Requiring DNS entries to be manually entered ▪ Separate group allocates security privileges ▪ On-premises connectivity ➢IaC depending on manual intervention cannot have automated tests ▪ Localize the manual intervention requirements ©2023 Derek C. Ashmore, All Rights Reserved 14
  • 15. Accommodating Manual Processes ©2023 Derek C. Ashmore, All Rights Reserved 15
  • 16. Infrastructure Code Testing ➢IaC is code! ▪ Housed in source control ▪ Often changed and released ▪ Needs testing like any other code ➢IaC change can have negative impact ▪ Environment outages ▪ End-user internet connectivity outage ▪ Application outages ▪ Testing team delayed for four days ➢Testing IaC can minimize negative impact ©2023 Derek C. Ashmore, All Rights Reserved 16
  • 17. Infrastructure Code Testing Differences ➢IaC != Application Code ▪ IaC requires external resources (e.g. Cloud) to run ▪ In-process unit testing often not possible ➢Limited localized (in-process) testing ▪ Generally limited to syntax checks ▪ Terraform validation ▪ Ansible Dry Runs ▪ IDE syntax checks ➢Most testing is “integration” testing ©2023 Derek C. Ashmore, All Rights Reserved 17
  • 18. Infrastructure Code Testing Challenges ➢Friction ▪ Harder to write/maintain ➢Dependencies ▪ Managed by other teams ➢Testing costs ▪ Use Sandbox – tear-down after tests ➢Manual intervention requirements ▪ Not possible to automate tests ©2023 Derek C. Ashmore, All Rights Reserved 18
  • 19. Lack of Discipline Causes… ➢Unplanned Work ▪ Change due to automatic upgrades ▪ Unintended consequences ➢Accidental over-writing changes of others ▪ Merge conflicts ▪ Changes deployed from unmerged branches ➢Increased defect rate ➢Configuration Drift caused by manual change ©2023 Derek C. Ashmore, All Rights Reserved 19
  • 20. Frequent Mistakes in the Field… ➢Lack of testing for common IaC code ▪ Testing for one use of common code is not sufficient! ▪ Often breaks other consumers of the common code ➢Creating a blast radius for IaC that’s too large ▪ Can’t make targeted changes without unintended consequences ➢Treating common IaC code as an enforcement mechanism ▪ Decouple policy enforcement and naming conventions ▪ Common IaC is a productivity enhancer only ©2023 Derek C. Ashmore, All Rights Reserved 20
  • 21. Agenda Intro and Level Set DevOps Best Practices DevOps Management Summary / Q&A ©2023 Derek C. Ashmore, All Rights Reserved 21
  • 22. Management is different too! • Instead of – Manual reviews/approvals • Automate guardrails • Automate testing • Whitelist cloud services • Consider continuous delivery/deployment – Capacity planning up front • Monitor cost increases and investigate – Mandating policy changes by edict • Automate policy enforcement ©2023 Derek C. Ashmore, All Rights Reserved
  • 23. Things that Don’t work • Adding Manual Approvals/Reviews – Kills velocity and productivity – Stops innovation – Creates bottlenecks • Forcing manual procedures – Attempt to “expedite” – Creates technical debt ©2023 Derek C. Ashmore, All Rights Reserved
  • 24. Things that Work! • Declare War on manual approvals – Favor automated guardrails – Automate oversight • Decentralize Cloud Management – Let app teams manage app infrastructure • Don’t be a bottleneck – Leave App teams to innovate • Create a Service Catalog – Automate whitelisting of services – Create reasonable process for new services • Legal reviews (HIPPA, GDPR, etc.) • Fund automation – You won’t make it manually! • DevOps Team Discipline – Automation needs SDLC just like applications – Source management is key ©2023 Derek C. Ashmore, All Rights Reserved
  • 25. Thank you! • Derek Ashmore: – Blog: www.derekashmore.com – LinkedIn: www.linkedin.com/in/derekashmore • Connect Invites from attendees welcome – Twitter: https://twitter.com/Derek_Ashmore – GitHub: https://github.com/Derek-Ashmore – Book: http://dvtpress.com/ • Please fill out the evaluation form! ©2023 Derek C. Ashmore, All Rights Reserved 25