Continuous Delivery in a Legacy Shop—One Step at a Time

DT3
Concurrent Session
11/12/15 1:30pm

“Continuous Delivery in a Legacy Shop—One
Step at a Time”

Presented by:
Gene Gotimer
Coveros, Inc.

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 · 904-278-0524 · info@techwell.com · www.techwell.com

Gene Gotimer
Coveros, Inc.
Gene Gotimer is a senior architect at Coveros, Inc., a software company that uses agile
methods to accelerate the delivery of secure, reliable software. As a consultant, Gene works
with his customers build software better, faster, and more securely by introducing agile
development and DevOps practices such as continuous integration, repeatable builds, unit
testing, automated functional testing, analysis tools, security scanning, and automated deploys.
He has successfully brought these techniques into commercial and government clients,
including the US Department of Defense. Gene feels strongly the repeatability, quality, and
security are all strongly intertwined; each of them is dependent on the other two, which just
makes DevOps that much more crucial to software development.

© Copyright 2015 Coveros, Inc. All rights reserved.
Continuous Delivery in a Legacy Shop -
One Step at a Time
Gene Gotimer gene.gotimer@coveros.com
Senior Architect @CoverosGene
2© Copyright 2015 Coveros, Inc. All rights reserved.
Coveros builds security-critical applications using agile methods.
Coveros Services
– Agile transformations
– Agile development and testing
– DevOps and continuous integration
– Application security analysis
Agile & Security training
Government qualifications
– DCAA approved rates and accounting
– TS facility clearance
About Coveros
Areas of Expertise

Selected Clients
The Problem

The Project
COTS product integration for DoD
– custom Python glue
Barely starting Agile
– Daily Stand-ups (really daily status calls)
– 2-week Sprints
– Good, pruned backlog
– No automated testing
– No unit tests
– No continuous integration
Releases every 6 months or so
– Freeze 2-4 weeks in advance
– Deploy Friday evening to Sunday afternoon
– Repair broken functionality Monday and Tuesday (and on)
The Delivery Team
Development – Local
– 2 Developers
– 1 Business Analyst
– 1 Project Manager
Test and Integration – Remote
– 4-6 Testers
– 4-6 Integrators (including security experts)
– 1 Information Assurance
Off-team
– Systems Administrators, hardware and software

The Problem
“But it works on my machine!”
The Problem
“But it works on my machine!”
=
High risk deploys

DevOps is
One Single Line of Code
“How long would it take your organization to deploy a change
that involves just one single line of code?
Do you do this on a repeatable, reliable basis?”
-- Mary and Tom Poppendieck
Implementing Lean Software Development: From Concept to Cash

Everybody is Responsible for Delivery
“The goal of DevOps is not just to increase the rate of change,
but to successfully deploy features into production without
causing chaos and disrupting other services, while quickly
detecting and correcting incidents when they occur.”
-- Gene Kim, Author of The Phoenix Project
Continuous Delivery
Make releasing a business decision, not a technical decision
High-confidence releases
– Small releases
– Fully tested
– No expectation of problems
Hotfix releases
– Possible
– No more than moderate risk
– No more than moderate coordination
Continuous Deployment was not a goal

The Approach
The Approach
Started with things that were in our control
– Dev and Test environments
– Development process
Make changes behind the scenes
– Free/open source tools
– Easy to integrate into our CI system
– Small changes
Disclose the changes when there was a win
– Highlight ease of use
– Use as justification for higher environments

The Journey
1: Continuous Integration
Trouble explaining “integration”
– between two or more developers
– not between systems
Just brought in SecureCI and wired to the ALM tool we had
– Jenkins (Hudson at the time)
– Nexus
– SonarQube (Sonar at the time)
– Automated builds
PMD
FindBugs
Checkstyle
Cobertura
Later added Python tools
Explained the advantages later

2: Functional Testing
Functional testing was done manually
– from a script written in Microsoft Word
We waited a year before staging a coup
– we didn’t want to encroach on their domain
Demo of Selenium
– demonstrated record-and-playback through the Selenium IDE
– we recorded the first set of tests
– turned it back over to the test team
They argued later that automated testing was ineffective
– the automated script (singular) only worked one time, then needed
to be re-recorded when any changes got made to the app
2: Functional Testing
We took it back
Rewrote existing tests in Java
Showed our business analyst how to clone-and-mutate the
Java tests
Started with JUnit, but went to TestNG
– better tagging and parameterization
– pre-test run initialization
Since it was COTS, focused on testing system interfaces,
not application functionality
Also began testing user roles
– Security testing = what can this type of user NOT do

Continuous Delivery
Project Manager came
across the book in a book
store
Everything made so much
sense
Logical extension of what we
were trying to do
Addressed a lot of the issues
we were running into
No money or time for an
effort, so we adopted it as
our long-term goal
3: Automated Deploys
Started with automating a Drupal web server install
– new system, not yet in production
– database server was easy, so we skipped it for now
Then automated the manual COTS install
Then started reverse engineering the broken COTS installer
Down the road, realized we could “Automate Everything”
No Puppet Enterprise Server
– just manually ran puppet apply from the command line

4: Security Testing
Tested Security Blanket (now by Raytheon)
– couldn’t get it purchased
We knew it couldn’t replace the “official” testing
– plus, didn’t want to encroach on their domain
Knew we had some good basis in our CI, static analysis,
and user role testing
Wanted a security scanner
– at the time, none worked with client certificates out of the box
Found w3af
– Python
– customizable
– client certificate support was there, but not exposed
– handed it over to the “experts” on the integration team
4: Security Testing
Found 0 vulnerabilities!
Perfectly secure! Yeah!

4: Security Testing
Never got past the login screen
But didn’t start at the beginning, so they even missed a XSS
bug on the home page
Never read the output or log
So we took it back
– Eventually had problems getting customized w3af to work properly
– Switched to OWASP ZAP, run manually
Security team focused on STIG and SELinux
– that was their expertise anyway
4+: Security Testing
Over a few days, implemented OpenSCAP in Jenkins for
STIG checks
– immediately found issues
– started adding Puppet manifests for remediation
Started using Nikto2 for web server scanning
– immediately found issues
Started running weekly scans of dev and test using
OpenVAS (Nessus fork)
– no immediate issues, but started seeing package security updates
before they became IAVMs
Discovered SELinux was in permissive mode
– had never been in enforcing

5: Performance
Reconfiguring database server
– we thought it was getting slower
Added JMeter to measure a baseline
Took rough breakdown of most common queries
Repeated as a 15-minute test
Monitored trend
Added similar testing to functional tests, another 15 mins
Also, number of functional tests was growing slowly
Watched functional test elapsed time as rough guide
6: Culture Clash
Continuous Delivery was being openly discussed
– PMO had just started thinking of it as a clear plan
– Kept asking when “continuous delivery” would be delivered, how it
would be packaged
Test and Integration started complaining
– 3½ of us were pushing the 12+ of them too hard
– moving too fast
– not a risk or control complaint, merely effort
People on test and integration team started leaving
– including “Burt”

The Results
Contract Renewal Time
Test and Integration decided not to renew
– all remaining personnel ended project with a month
Security issue found the following week
– deployed 3 days later
Went back to 2-week deploy cycles, sometimes faster
Left 3 people on development team
– One went back to take over for the test and integration team as
hands-on-keyboard
– BA left project and another came in ½ time for testing
Dropped into maintenance mode

The Delivery Team
Development – Local
– 1 Developer
– 1 Release Manager
– ½ time Tester
Test and Integration – Remote
– 1 Information Assurance
Off-team
– Systems Administrators, hardware and software
The Project
Barely Agile
– Maintenance only
– Kanban-ish, tracking WIP
– Daily Stand-ups (really daily status calls)
– 2-week Sprints
Releases prepared every 2 weeks
– Soft freeze Thursday for Friday release
– Deploy Friday evening
– 100% working functionality Friday evening
– Non-event
Configuration parameters
– from 200+ untracked values
– to ~30 Hiera-controlled values
Biggest coordination issue: 72 hours for user messaging
Biggest time consumer: 3-6 hours for VM clones

Missed Opportunities
Missed Opportunities
Automated deploys
– more valuable than just reducing risk
Vagrant
Some security scanning earlier
– can’t assume someone else is doing it
Some performance testing earlier
We relied on client-side certificates for authentication
– EJBCA should have been set up immediately
Upgrades are a huge time sink
– components, libraries, applications, system software
– add tools to track it as early as possible

The Tool Chain
The Tool Chain
Jenkins
Puppet (no Puppet Enterprise)
– 2 puppet apply commands per server
one --noop for system audit
one for deploy
Security
– OpenSCAP (every deploy, minutes)
– OpenVAS (every weekend, hours)
included Nikto2
used Kali Linux
– OWASP Dependency Check (on-demand, many minutes)
– OWASP Zed Attack Proxy (on-demand, few days)
– Full role-based Selenium test coverage (every deploy, overnight)
10k+ Selenium tests, TestNG parameterized

The Tool Chain
Testing
– TestNG for Java unit tests
– Nose for Python unit tests
– Mockito/Mockito for Python
JMeter
– for some representative performance tests
Static Analysis
– Java
PMD
FindBugs
Checkstyle
Cobertura
SonarQube
– Python
Pylint
coverage.py
Questions?
Gene Gotimer
gene.gotimer@coveros.com
@CoverosGene

Continuous Delivery in a Legacy Shop—One Step at a Time

More Related Content

What's hot (20)

Viewers also liked (7)

Similar to Continuous Delivery in a Legacy Shop—One Step at a Time (20)

More from TechWell (20)

Recently uploaded (20)

Continuous Delivery in a Legacy Shop—One Step at a Time