Large scale agile development practices

EXPERIENCE REPORT:
LARGE SCALE AGILE
DEVELOPMENT PRACTICES
Daphne Chong (@daphnechong)
Niall Connaughton (@nconnaughton)

What was the project?
 C# Winforms
 SQL Server database
 Large CRUD system
 App deployed to client sites
 Started in 2001
 Market leading product

What was the project?
 Low volume
 Performance important, but not critical
 Goal was a stable, consistent, feature-rich app
 Rapid release cycle (after any successful build)

In 2006…
 60 Developers
 5 million lines of code (including tests)
 ~100 solutions, 250 projects
 300+ db tables
 160,000 automated tests

Today…
 60 Developers
 ~10 million lines of code
 485 solutions
 470 db tables
 343,090 automated tests

The Pit of Success
We want our customers to simply fall into winning
practices by using our platform and frameworks
– Rico Mariani
Build platforms [where] developers just fall into
doing the “right thing”... Types should be defined
with a clear contact that communicates effectively
how they are to be used (and how not to)
– Brad Abrams

Pit of Success – Technical
Factors
 Inheritance
 Template Method pattern
 Strict naming conventions
 Common reusable controls
 Own type system
 Single generic repository – handled common
scenarios in CRUD, transactions, referential
integrity, caching, persistence

Pit of Success
 Tried to avoid too much divergence in code
 Made it difficult for people to stray from the “true
path”

Consistency
 70 % of a dev’s time is understanding existing code
– Peter Hallam [2]
 If you can reduce that even by a small amount, it’s
significant productivity gain

Consistency
 Consistent approach to everything
 Naming conventions – project, solutions, file locations,
namespaces, class names, methods, database tables
and columns
 Formatting
 UI look and feel (achieved through base controls)
 Patterns of design

Consistency
 Standardised on a core set of technologies that
best fit the application
 Avoided “technology soup” support and
maintenance
 Reduced learning curve for new developers

Insane things we did
 60 devs, no feature branches!
 SourceSafe 
 Highly sensitive to the build breaking
 True continuous integration
 Mitigated by clients opting in to “alpha”, “beta” or
“stable” versions
 Two large re-writes of architecture
 Result was more suited to our needs than if we had
kept the first and refactored it progressively

How did we build?
 Full build from batch file
 Update source files
 Solutions built in dependency order defined in xml
 All binaries output to one folder
 Build on dev machine = 20-25 minutes
 Build server with RAID = 4 minutes
 Quick build tool (written in c#)
 Only build solutions that have changes

Development - Architecture
 Built our own architecture
 Re-usable controls
 Comprehensive base classes
 Own type system
 Primitive types as well as Custom types
 Could define helper methods .IsEmpty,
.SubstringSafe() , IsInRange()

Development - Architecture
 Architecture constantly extended
 Approach was always unified though the
architecture
 Enabled good ideas to have widespread effect

Development – Technical debt
 Management support for improving bad code
 Aggressive attitude to technical debt
 Very little dead code
 Example: methods with 7 overloads that can’t be
deleted because don’t know who’s consuming them

Development – Technical debt
 Constant small amount of pain
 Rare occurrences of large amounts of pain
 Once a hard cost has been paid, make sure that
nobody ever has to pay the cost again

Development - Custom Tools
 Automated repetitive and error prone tasks
 Custom ORM generator
 Database upgrading tool
 GUI binding helpers to ensure targets exist
 Build monitor system tray application
 Code Generators for new modules
 Automated deployment tool

Testing
 Two types of tests
 Business Tests: user functionality
 Meta Tests: code design and contracts

Testing
 Most of our tests were actually integration tests of
varying scope
 Didn’t use a mocking framework (not yet well
known)
 Created our own stubs by inheriting from the
concrete class and overriding any methods that
used resources, etc.

Business Tests
 Generally state based logic
 Easy to read and write
 Fairly resilient throughout refactoring
 Derived from a base TestCase with ~25 basic tests
 Array properties don’t return null
 Properties don’t throw NullReferenceExceptions
 Don’t alter the state of the object before shown in GUI

Meta Tests
 Don’t test direct functionality
 Enforced consistency, patterns and code contracts
 Implemented using reflection and attributes
 Slow to run, but very valuable

Meta Tests – Examples
 Check code conforms to conventions and contracts
 Public constructor with a specific parameter exists
 Assemblies are strong named
 Check for memory leaks
 Check that each model had an associated test
[TestsSubclassesOf(typeof(BusinessBase))]
public abstract class BusinessBaseTest
{
// contract tests here
}

Meta Tests
 The Meta Tests help to enforce the rules, rather
than having a “bad cop” in your team
 Not political or personal
 Once established, it gave power to the rest of the
team to contribute new tests and rules

Meta Tests
 Allowed the architecture to grow rapidly
 Already know the code conforms to a particular
standard

How do you run 343,000 tests?
 Initial approach similar to CruiseControl
 One machine building and running all tests
 Results emailed after completion
 … stalled once we hit 25,000 tests
 … Took 4 hrs to run
 Feedback cycle wasn’t fast enough

Distributed Testing Framework
 Highly customised
 Tests distributed to up to 60 agents
 Build time around 1 hour
 if built on single agent = 30-40 hours
 Package on successful build
 Agents ran on idle developer machines
 Ran 24/7

Distributed Testing Framework
 Developer submitted builds could be processed
(i.e. pre-checkin)
 Task tray to signal progress
 Red light = broken tests, broken build or checkin
embargo
 Green light = OK to check in

Team
 Very Flat structure
 6-8 devs in architecture team (“Blue Code”)
 Business & GUI architecture
 Development tools
 Testing framework
 Database development
 Upgrading code
 Most other developers building business rules into
software (“Red Code”)

Architecture vs Business teams

Team - Architecture
 Made up less than 20% of team
 Improved productivity by much more than 20%
 Allowed devs to focus on smaller slice of stack and
get quicker feedback cycles

Team - Business
 Low barrier to entry for development
 Junior or new developers could be productive very
quickly
 Varied skill sets in the team were utilitsed
accordingly
 Technically minded people drawn to architecture
 Business minded people focused on implementing
customer requirements

Team – Sharing Information
 Free flowing collaboration
 Some development overhead in helping other
people
 Some information silo-ing

Team – Pair Programming
 Initially strict pairing
 Became “organic”
 Not enforced
 Never discouraged

Team – Work Ethic
 People took a lot of responsibility and
accountability of their own accord
 Boy scout rule – leave things cleaner than when
you found it
 Always fix broken windows

Work satisfaction
 Low turnover rates for first 4 years
 Lots of smart people to work with
 More meritocracy than political
 Developers took pride in their work
 Ability to implement large scale change open to
everyone

Work Satisfaction – flip side
 Hard to have interesting work for 60 developers
 Could be pigeon-holed into the business layer
 More stress than an average agile environment
 Long work hours implied because there’s only one
build – responsibility to keep the dev machine
going. No leaving the build broken or failing tests
throughout lunch or overnight.

Conclusion
 Team made a lot of investment in design and
conformity
 Easy to do with a CRUD system, harder with other
types of business software
 Project has expanded and moved on, but the
architecture has evolved with it and is still in use

Thanks to…
Zubin Appoo
Senior Manager, Product Development
www.cargowise.com

References
 [1] Brad Abrams, The Pit of Success
http://blogs.msdn.com/b/brada/archive/2003/10/02/50420.aspx
 [2] Peter Hallam, What do Programmers Really do Anyway?
http://blogs.msdn.com/b/peterhal/archive/2006/01/04/509302.aspx

Large scale agile development practices

More Related Content

What's hot (20)

Viewers also liked (11)

Similar to Large scale agile development practices (20)

More from Skills Matter (20)

Recently uploaded (20)

Large scale agile development practices

Editor's Notes