Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Testing within an agile environment
Chris Gollop & Beyza Sakir

Overview
History
Changing Testing Concerns
Adaptations
Future
Conclusion
Further Reading

History – what we do
Release every 2 weeks
Trade $billions/day
Average 0.4ms execution time for MTF members
10,000 orders second
1 tester per 5-6 developers
Very low rate of production issues

History - company
Conceived as an Agile company
Agile has matured but we’re still adapting
Agile testing:
Acceptance Tests
Integration Tests
Exploratory Testing

History - company
•This has allowed us to experiment and adapt:
•Time freed up
•Quick feedback on changes
•Historic data available to analyse

History – test team
4 test specialists with different strengths complement each other as a 'team‘
Rotate through all teams
•No Test Manager
•Part of development team though test interests have a separate iteration
meeting
•Testing is responsibility of everyone

History – team structure
l
Team One Team Two Team Three
Development Specialists
Business Specialists
Test Specialists

History - innovate
Keep changing, even if something appears to work there may be something
better
“If I had asked people what they wanted, they would have said faster horses. I
gave them a car” (Henry Ford)

History – measure and feedback
Measure
To know if it is effective
Many monitoring and measuring tools
Feedback
Tying feedback to ideas
Quicker this is the faster you can confidently adapt
Can be numerical or social

Changing Testing Concerns Over Time
Traditional Testing Maturity Models start at “ad-hoc” and move through
structured testing before finishing at “Continuous Improvement”
Some of our test concerns (over time)
Decrease Manual Testing & increase automated Acceptance Tests
Performance
DSL
Speed of Feedback
Increasing number of Edge Cases as system becomes more complex

10,000 Behavioural Tests
8,700 Acceptance Tests (UI, API)
1,300 Integration Tests (subsets of services using internal interfaces)
Acceptance Test would test a rejection
Integration Tests would test the different rejection types

Was right at the time as needed Acceptance Tests to get good coverage
quickly, trade-off is they are slower to run repeatedly.
Focus is now quicker feedback.

Adaptations – Fitness Landscape

Adaptations – Fitness Landscape
Each square is similar to it's neighbour
Higher it is the better the solution – valleys are bad, mountaintops are good
You don't fully know where you are on the landscape
You feel your way by trying out new ideas and recognising failure
You may be half-way up the mountain rather than at the top
And the mountains move!
Some peaks move faster than others:
Google v McDonalds
Evolution isn't just biological it's everywhere

Adaptations – Palchinsky Principles
Peter Palchinsky was a Russian engineer who advised on (against) Lenin Dam and
Magnitogorsk – large projects which essentially failed due to being too big and unable
to adapt.
Palchinsky Principles are shaped to encourage stronger innovation, better leadership
and more effective policies:
Variation - seek out new ideas and try new ideas
Survivability - when trying something new do it on a scale where failure is
survivable
Selection - seek out feedback and learn from mistakes as you go along, avoid an
instinctive reaction of denial

Adaptations – Recognise failure quickly
Being able to recognize a failure means you can re-cast it into something more likely to
succeed.
Accept something is failing, don’t deny or ignore it.
"Hard to see what changes would improve current process when they’re currently
sweeping all problems under the ‘our context’ rug." (Anthony Green)
The essential first step is being willing to fail.

Adaptations - Google v Apple
Both considered as innovative companies.
Apple innovates in things that are core to its business
Google innovates on things that indirectly contribute to its business
Apple releases products that are polished and well-scoped
Google releases many different products (Buzz, Wave, Docs, AppEngine) early in their
development and lets the community develop and adapt them.
Apple create new mountains on the fitness landscape
Google is surfing along on, or near, the moving fitness landscape mountain-top by
adapting its strategy and offerings.

Adaptations – a brief history of what we have tried

Adaptations – what worked
Production Data
Testing in Live
Big Feedback
Remove Intermittent Tests
Tester Pairing

Adaptations – Sanitised Production Data
Raw data (database)
Patterns of use
Data profile patterns
Migration (automated)
Real world data
Larger than we can create
Many edge cases
Try and reincorporate into our Acceptance Tests
Performance
Staging
Progression in what and how we use the data

Adaptations – Testing in Live
Uses same hardware and code as it is just an isolated venue
Runs a subset of the Acceptance Tests
When we expanded our Exchange by adding a multi-tenanted concept known
as a venue we took advantage for testing
Verifies that the various components of the exchange are hooked up and
configured correctly and that the deployment process worked correctly
Runs automatically throughout the day as one of our monitoring systems

Adaptations – Big Feedback
Important CI pipeline builds at the top
Red (failed) items rise to the top
Continuously and easily expanded
Useful features survive
Failure ownership/failures at a glance - AutoTrish
Intermittent tests - Magic Eight Ball
Auto Embargo
Performance metrics
Build sheriff to keep check-in embargo

Adaptations – Intermittent Tests
Tests that can fail even if feature is working
Now have <1 intermittent tests per run (out of 8,200)
Each about 1 test run in 20
Hides genuine failures
Have previously missed a release as couldn't tell if genuine
Can result from
Asynchronous messaging
Tests changing global settings/lack of isolation
End of day transitions
Date/time

anon
anon

All Failures run twice
Fail twice = failure
Fail once, pass once = intermittent
Accepted as something we need to deal with
Work to remove
DSL improvement
Fix a genuine issue (race condition)
End of iteration is given over to removing intermittency
Build Sheriff also works on this

Adaptations – Tester Pairing
Each tester has their own strengths
Noticed that at rotation handover the new tester generally found a bug
Issue with the testers (denial)
Something we can harness (acceptance)
Already do development pairing (Dev & Dev and Dev & Tester)
When do we pair?
At the right time for the story
Initial brainstorming when creating a mindmap
When we feel we need to (pull)
When someone has time (push)

Adaptations – what didn't work
Not many absolute failures
Although many reasons it can fail
Even if something fails you learn something
Natural selection will continue
You can't stay still, unless you are improving you are losing ground as others
will innovate or adapt

Adaptations – facts
Of the top 25 industrial corporations in the United States in 1900, only two
remained on that list at the start of the 1960s. And of the top 25 companies on
the Fortune 500 in 1961, only six remain there today (IBM)
Average life expectancy of a Fortune 500 company has declined from around
75 years half a century ago to less than 15 years today, and heading towards 5
years (Deloitte)

Future
Expanding the way we test
Speed up feedback
Increasing integration test coverage
Pushing UI to API – testing at the right layer
Change AT run order
Improving feedback/communication
Test team standups
Documentation
What is just enough
How to get people up to speed in a complex system

Future
Is about the people
Team make-up is varied and we recruit to keep it this way
“’Testing is a strictly formal process that by its nature should not vary’ - this is a
typical belief by people who know nothing about testing.” (James Bach).

Conclusion
Need to experiment and adapt to stay ahead
Even if something works it does not mean there isn't something better
Make changes on a scale where failure can be survived
Do not be afraid of failure but know how to recognise it quickly

Further Reading
•Tim Harford
•Richard Wiseman
•Malcolm Gladwell
•Daniel Kahneman
•Dan Ariely
•Richard Thaler
•LMAX Staff blogs - http://blogs.lmax.com/staff-blogs/

Testing within an Agile Environment - Beyza Sakir and Chris Gollop

Testing within an Agile Environment - Beyza Sakir and Chris Gollop

More Related Content

What's hot (20)

Viewers also liked (18)

Similar to Testing within an Agile Environment - Beyza Sakir and Chris Gollop (20)

More from JAXLondon2014 (19)

Recently uploaded (20)

Testing within an Agile Environment - Beyza Sakir and Chris Gollop