Agile DevOps Transformation At HUD (AgileDC 2017)

www.coveros.com
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 1© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED.
Agile DevOps Transformation At
HUD
AgileDC 2017

www.coveros.com
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 2
www.coveros.com
Agenda
• About Me
• Introduction
• Technology Overview
• What Is DevOps?
• Our Approach To DevOps
• How We Took An Application To Production in 7 Months
• Lessons Learned

About Me
• Marco Corona
•marco.corona@coveros.com
•@MarcoCorona92
• Graduated from Allegheny College in 2014
• Consultant at Coveros since 2014
• Currently - DevOps (CI/CD) for HUD
• Most Lombardi Trophies in the company
Place your photo here

Introduction

• Coveros helps companies improve the agility and security of their software
applications, teams, and enterprise
• Consulting Services
• Agile Transformations
• Secure Agile Development & Testing
• DevOps Implementations
• Agile Enterprise and Team Coaching
• Application Security Assessments
• Agile, DevOps, Security Training
• SecureCI – Open source CI/CD
tool stack
About Coveros
Introduction
Areas of Expertise

Introduction
Selected Coveros Clients

• Housing and Urban Development
• “HUD's mission to promote homeownership, support community development, and
increase access to affordable housing, free from discrimination.”
• Informally they want to end homelessness, make homeownership affordable for all, and
strengthen communities.
• Budget is about 45 billion
• ~300 million is set aside for IT expenditures.
• Most HUD assets are used for maintaining housing capital, rental/housing
assistance/community outreach.
HUD
Introduction

• HUD wanted a pilot program showcasing the Agile/DevOps/CICD across several
legacy mainframe projects (10 to start)
• Has since been bumped to 15 projects
• Mainly Java and Javascript projects
• No Agile/DevOps/CICD in place
• No backlog
• All of HUD was steeped in Waterfall
• Software, management, and support
• Would take weeks for a virtual machine to be provisioned
• Would take years for an application to get into production
• Each application was developed by a different development team
The Project
Introduction

• CI/CD Team
• Project Manager, Tech Lead, 7 DevOps Engineers
• Control lower environments (dev/test)
• Separate team controls higher environments (staging/production)
• Our scripts are used to create and deploy to all environments
• Use SCRUM approach
• 1 week sprints
• Plan about 80% of work
• Also use Kanban
• Allow dev teams to open JIRA tickets for support
The Delivery Team
Introduction

•Azure - Cloud
•Azure Cli - Infrastructure as Code
•Git - Version Control System
•Jenkins - CI Server
•Nexus - Artifact Repository
•Chef - Infrastructure as Code
•SonarQube - Code quality and Dashboarding
•Confluence - Wiki
•Jira - Agile Planning and Tracking
•Lots of Java, Mule, OpenAM, and Nodejs
HUD Infrastructure
The Technology Overview

• Owned By Microsoft
• Second largest cloud
•By an order of magnitude.
•47% vs 10% estimated market share.
•Azure CLI to enable automation
•Started with 2 subscriptions
•External subscription is where non PIVed work is done.
•External has since been decommissioned
•~1000 VMs to support the effort for all 15 existing development projects.
Cloud - Azure
The Technology Overview

Our Approach

“DevOps (a portmanteau of "development" and "operations") is a software
development method that stresses communication, collaboration and integration
between software developers and Information Technology (IT) professionals.” --
Wikipedia
DevOps is defined by and encompasses:
• Goals and Values
• Methods
• Practices
Definition of DevOps
What Is DevOps

• Findings from the 2016 State of DevOps Report:
• High performing organizations:
• Deploy 200 times more frequently
• Lead times are 2,555 times faster
• Recover from failures 24 times faster
• Are 2.2 times more likely to recommend their organization
• Spend 22 percent less time on unplanned work and rework
• Spend 50 percent less time remediating security issues
• Improving entire product lifecycle (planning, quality and security assurance, customer
feedback) speeds up delivery while improving quality, security, and business outcomes
DevOps Business Value and Benefits
What Is DevOps
2016 State of DevOps Report, Puppet Labs
https://puppet.com/resources/whitepaper/2016-state-of-devops-report

• Repeatable and Reliable Software Delivery
• Automate Everything
• Build Quality In
• Reduce Risk
• Keep everything in Version Control
• Small, Frequent Releases
• Production readiness is always a known state
• Definition of Done
• Collective Ownership - everyone is responsible for the Delivery Process
• Collaboration
• Continuous Communication
• Unified Processes
• Unified Tooling
• Continuous Improvement “Kaizen”
DevOps Goals
What Is DevOps

• Automate onboarding for dev teams
• Development workstations
• Automate the teams build process
• Use maven/gradle to build through jenkins
• Automate environment creation
• Azure cli and chef
• Automate deploys
• Chef and jenkins
• If you do it more than once, automate it
• Share code
• Azure, Chef, Jenkins, Maven/Gradle
High Level
Our Approach To DevOps

• Needed to get development teams to buy in
• DevOps is different
• Some viewed us as a blocker
• Some upfront overhead
• Develop a trusting relationship with development team
• Need to make sure they understand we are here to help
• 4 hour developer onboarding class
• Additional training when required/requested
• Teams are remote, can’t rely strictly on email/slack
• Share code
• Azure, Chef, Jenkins, Maven
Keys To Success

• Set up a standardized “Hello World” pipeline
• Idea was for all new projects to use this as a template
• Served as a “best practices” repository
• For build code, jenkins jobs, chef code
• Simple Java project that deployed to Tomcat
• Built using maven and gradle
• Deployed using chef and jenkins
• Ran security scans
• Set up unit test dashboards
• Serves as a good starting point to onboard projects
• Trained the team
Keys To Success

Continuous Integration in a DevOps Culture
Continuous integration is the act of frequently integrating different developer’s code, building and testing each commit to
find any problems quickly.
The goal is that software is always in a working state.

• Automate their build process locally
• Using maven/gradle
• Replicate build process in Jenkins
• Build and run unit tests on every commit
• Dependent on git branching scheme
• Start collecting unit test metrics and push them to SonarQube
Continuous Integration
Our Approach

• Goal: Software is deployable to production at all times
• Delivery Pipeline
• assess and give feedback on the state of the software throughout the development process
• Automation
• make the deploys repeatable and reliable
• Testing
• remove the risks of deploying software
• Deploying software to production becomes a business decision
Philosophy and Mindset
Continuous Delivery

• Automate creation of virtual machines
• Was very difficult in the early days because of Azure’s immaturity and our lack of expertise
• Automate software installation
• Work with development team to manually deploy application
• Very important to see the steps
• Allows us to build a greater relationship with the team
• Automate the application deploy
Continuous Delivery
Our Approach

• Built into our pipeline, so it is not an afterthought
• Application Security
• SonarQube runs static code analysis with each build
• Findbugs, PMD
• Yasca runs with each build
• Open source tool written in PHP
• Easy to create your own rules
• Nightly Fortify runs
• Environment Security
• Nightly OpenSCAP runs
• Mitigate security concerns through chef
• All scans’ results uploaded to sonar
Built-in Security
Our Approach

• Occurs once the application is deployed
• Must be quick
• Tests the deployment, not the functionality
• Focus on
• Basic signs of life
• Interfaces between systems
• Configuration settings
Smoke Testing
Our Approach

• Quality Focused
• Test until confident that you have a viable production release candidate
• Types of tests:
• Functional tests
• Regression tests
• Performance testing
• 508 compliance tests
• System integration tests
Acceptance Testing
Our Approach

Implementation

• Project with most internal visibility
• Used by all of HUD
• Largest development team we have worked with
• ~20 developers
• Largest application with most dependencies
• Had a very strict timeline
• About 7 months from start to “go live” date
Background
Loan Review System (LRS)

• Java application with a Javascript frontend
• Deployed to JBoss
• Microsoft SQL Server for database
• Had no build automation
• Were building manually
• Had little to no knowledge of build automation tools
• Had no set architecture
• We had to provide them with an architectural diagram
• Were adverse to DevOps
• They saw us as a blocker
• Needed to get them to buy in to be successful
Getting Started

• Used Git
• Used feature branching
• Designated “dev-*” should be deployed to temp
• Designated “ci-sprint/dev” should be deployed to dev
• After testing is complete, merge back into master
• Designated “master” should be deployed to test
Branching Strategy

• Team chose Maven as their build tool but had no knowledge
• Very messy and often incorrect Maven code
• We were unable to build on Jenkins
• Had to go in and refactor their build code without impacting development
• Added a “jenkins” profile
• Versioning the artifacts became an issue
• Format 1.0.${Jenkins Build Number}
• Maven does not like dynamic versioning
• Worked on jenkins, but not locally
• Came up with a workaround with a groovy script
Automating The Build

• Deployed a front end and a back end
• Deploy to JBoss through chef
• Had a specific JBoss configuration file
• App was developed on windows but deployed to linux
• Local environment was setup manually, led to “works on my machine”
• There were many environment specific properties
• These properties were externalized into a properties file
• This properties file was managed by chef
• Allowed us to change the value based on environment
Automating the Application Deploy

• We suggested liquibase to manage schema changes
• Liquibase uses XML, YAML, JSON, or SQL- vendor independent
• Rollbacks are handled as refactoring
• Conditional execution
• Can also be used to seed test data
• They had no expertise to liquibase so it took a while to get this in place
• For the first couple of months, we had a database backup in nexus that we
deployed each time
• This made me feel gross
Automating the Database Deploy

● Each build with “dev-*” pattern gets its own dynamic functional test environment (FT)
● If build is successful, FT does not get destroyed for 12 hours
o Allows teams to debug, manually test, etc.
● Build and FT Creation happen in parallel
● If build fails, the FT gets put into a database for 1 hour
o If the FT is not pulled by a job within an hour it is destroyed
“Dev-Flow” Pipeline - High Level Requirements

● Branch off master
● Develop code
● Push code to remote branch
● Jenkins picks up the SCM change and builds the code
● Build artifacts are stored in Nexus
● Build artifacts are deployed to dev/test environment through Chef
● Theoretically, tests are run
Standard Pipeline Workflow

• Deploy every two weeks
• 1 sprint
• Deploy to Staging Wednesday
• Deploy to Production Friday 8pm
• This is a very low risk deploy since our deploy process has been practiced many times
• Also the code has been thoroughly tested in dev, test, and staging
• Do not rebuild code
• Use artifacts that are already in Nexus
• Submit Release Document to Cloud Team
• Old HUD documents were 10-20 pages
• Now they include 1 step
• “That was the least painful go live we’ve ever had”
Promoting Code to HIgh Environments

• DevOps is a culture
• Can have the best tools and processes in the world, if no one follows them, they’re useless
• Getting buy in from all parties is critical
• Never stop improving
• Keep improving your pipeline and making it more robust
• Automate simple tasks so you can spend time on more complicated tasks
• Technical Debt will happen
• Mitigate when possible
• Track it
Lessons Learned

Agile DevOps Transformation At HUD (AgileDC 2017)

More Related Content

What's hot (20)

Similar to Agile DevOps Transformation At HUD (AgileDC 2017) (20)

Recently uploaded (20)

Agile DevOps Transformation At HUD (AgileDC 2017)