Improving Quality through Continuous Integration - A case study of CollabNet

1 Copyright ©2014 CollabNet, Inc. All Rights Reserved.ENTERPRISE CLOUD DEVELOPMENT
Improving Quality
through
Continuous Integration
A case study of CollabNet best practices
Janardhanam Venkat
Director Engineering
Dr. Dobbs Conference - Pune - April 13th 2014

2 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
Overview
• Introduction to Basic CI
• Chaos in Product Development
• Defining CI strategy
• Building a CI strategy
• Execution cycle feedback
• CI Infrastructure
• Drill down to Metrics
• Measurement & Maturity

CI Packaging
CI Server
Ping
Version
Control
CI Server
Commit
- Detailed Test Suite
Longer Running Test
Light weight Test
Basic CI Environment
- Unit Test
- Check Style

Chaos & Problem
Where to start?

• Not sure how much unit test coverage is written for a feature
• Not sure if my code is following best practices
• No time to test in multiple browsers and their versions
• Unable to cope up with installer testing
– Combination of OS & database has many permutations to test
– Turn around time to test a minor installer fix is 2 weeks or longer
Agile Team Problem

• Infrastructure Security Vulnerability
– Prior Product release versions in production
• Application Security Vulnerability
– Cross site scripting, SQL injection
• Service Issues
– SOAP API breaking
• Broken Link in the product
– Help contents 404
• Few feature have issue that are browser specific
– Not working in IE 9
Customer Problem
Reactive
(Customer
Problem)
Proactive
(Agile Teams
Problem)

• Use provisioning to enable CI
• Run parallel tests
• Tie code review process with commit tests
• Test in clone of Production environment
Industry Best Practices

Input to CI Strategy
CI
Strategy
Customer
Problem
Agile Team
Problem
Industry
Best
Practices
Roadmap

• Parameters that defines the strategy
– What kind of test Coverage?
• Certain features may need more coverage
– How important is Time to Feedback?
• Can team act upon if the feedback takes longer
– What is the Cost?
• Hardware versus Tester Resource
• Pick license or Open Source tool
Refining CI strategy
Source: Revello Systems, March 2014
You may not get all three parameters right

Building CI strategy
Problem Tools Value
Infrastructure Security Vulnerability Nessus Identify JDK, Apache,
OS vulnerability
Application Security Testing Acunetix XSS, SQL injection
Installer Automation Home grown Multi platform testing
Functional Testing Selenium Reduce time
Services Testing SOAP UI Automate
Static Code Analysis FindBugs, PMD Code Quality
Broken Links Link Checker Identify 404
Code Coverage Cobertura Unit test coverage
Performance Testing JMeter Load Testing
Browser Compatibility Testing Browsera Multi browser test
508 Compliance Testing WAT

Extending CI Infrastructure
Self
Provisioning
ALM
Jenkins
CI
Infrastructure

Leverage Self Provisioning Environment
• Provisioning of boxes
– Platform, OS & application configuration
– E.g.: Solaris, CentOS, Suse, RHEL, in combination with 32/64 Bit, 2/4/8
core & different storage space
– Common Profile for teams
Datacenter
Distributed Teams
SDLC Mgmt.
Customer
(Early release)
Product – Ops 1
Product – QA 1
Product – QA 2
Product – QA 3
Product – Dev 1

Leverage ALM for Traceability

CI Master
Nessus
Suite
Suite 1
Suite 2
…
Suite 6
CI Master
Detail Test
Suite
Suite 1
Suite 2
….
Suite 10
Commit Test
CI Master
Functional
Test
CI Master
SOAP Broken Link
CI Master
Installer
Automation
Case 1
Case 2
……
Case 14
FindBug PMD
CI Infrastructure – Self Provisioning, ALM & Jenkins
Boxes
Provisioned in
private cloud
Master
Slave
Around 70 Servers
ALM

CI Execution Cycle – Feedback
Commit
Validation
SOAP
Services
Test
Static
Code
Analysis
Functional
Test
Installer
Automation
Application
Security
Testing
Infra.
Security
4 Hr 1 Week10 Min 20 Min 3 Hr 10 Hr1 Hr
Detail
Unit Test
Commit reverted for failed test
Defect created for failed test
15 Min0

CI Execution Cycle – Feedback
Commit
Validation
SOAP
Services
Test
Static
Code
Analysis
Functional
Test
Installer
Automation
Application
Security
Testing
Infra.
Security
4 Hr 1 Week10 Min 20 Min 3 Hr 10 Hr1 Hr
Detail
Unit Test
Commit reverted for failed test
Defect created for failed test
15 Min0
598
1327
5601
5287
93 8
0
1000
2000
3000
4000
5000
6000
Commit
Validation
SOAP Selenium Full Unit
Test
Load Test Installer
Automation
#ofTestCases
TeamForge Automated Cases

CI Visibility Dashboard
Took an year to implement the strategy
Started in April 2013

CI Visibility Notification

Metrics - Customer Reported Core Product Defects
17.20
15.13
7.00
0.00
2.00
4.00
6.00
8.00
10.00
12.00
14.00
16.00
18.00
20.00
Jun-12 Dec-12 Jun-13 Dec-13
TeamForge Release Date
Customer Defect By Release Ratio

Metrics - Installer Automation
• Reduce Installer qualification time to 1 day – Phase I
– Total 185 hours to qualify one round of installer testing
– After automating 8 installer cases, reduced qualification time to 134 hours
Manual, 131
hrs
Saved, 51 hrs
Actual, 3 hrs.
Automated,
54
Installer Automation - 8 of 19 cases
done
One round manual qualification - Total 185 hours

• CI increases confidence and reduces risk
• Defines quality, prevention & reduction of defect
• Builds value with faster feedback
• Shorten software release time
• Results published, increases visibility to agile teams
• Challenges Continues
– Unable to automate data warehouse ETL testing
– Keeping up with tools & maintenance overhead
– Large scale refactoring & architecture changes
CI Value to Agile Teams

• Infrastructure vulnerability - Setup in prior 3 releases
• Broken link - Set up in prior product release
• Setting up the same CI infrastructure for 3 branches
Additional Refinements

Measure & Mature CI Strategy
Basic
Find defect after
development
Manual
Deployment
Team actions on
report
Commit
Validation
Manual
Build
Intermediate
Early detection
Partial auto
deploy to Test
Historical &
analysis
Code analysis,
Functional Test
Automated Build
& CI
Advanced
Reduce customer
defect
Auto deploy to
Test /Stage
Trending Reports
Performance,
Security Test
Leverage
provisioning
Optimized
Defect
prevention
Auto deploy to
Prod
Predictive
Modeling
Services Test
Track CI config
changesBuild
Test
Visibility
Deployment
Quality

24 Copyright ©2014 CollabNet, Inc. All Rights Reserved.24 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
Questions?
Janardhanam Venkat
[venkatj@collab.net]
www.collab.net
+1-650-228-2500
+1-888-778-9793
blogs.collab.net
twitter.com/collabnet
www.facebook.com/collabnet
www.linkedin.com/company/collabnet-inc

Improving Quality through Continuous Integration - A case study of CollabNet

More Related Content

What's hot (20)

Viewers also liked (13)

Similar to Improving Quality through Continuous Integration - A case study of CollabNet (20)

More from Venkat Janardhanam, MS, MBA (11)

Recently uploaded (20)

Improving Quality through Continuous Integration - A case study of CollabNet