The Evolution of Testing Methodology at AWS: From Status Quo to Formal Methods with TLA+
3 likes1,764 views
The document summarizes the evolution of testing methodology at AWS from a status quo approach to using formal methods with TLA+. It describes AWS services experiencing exponential growth and the need for better testing of distributed algorithms. It outlines their general test strategy and limitations of status quo test adequacy criteria. The document then details AWS' adoption of generative testing, in-process clusters, informal proofs, and formal specification with TLA+ to validate algorithm correctness. Real-world examples applying these methods to DynamoDB and other AWS services are provided.
![S3 experienced exponential growth for
6 years to reach 1 trillion objects
stored; less than a year later it reached
2 trillion objects [1]
5](https://image.slidesharecdn.com/untitled-150223093544-conversion-gate02/85/The-Evolution-of-Testing-Methodology-at-AWS-From-Status-Quo-to-Formal-Methods-with-TLA-7-320.jpg)
![DynamoDB processes millions of
transactions per second in a single
AWS region around the clock [2]
6](https://image.slidesharecdn.com/untitled-150223093544-conversion-gate02/85/The-Evolution-of-Testing-Methodology-at-AWS-From-Status-Quo-to-Formal-Methods-with-TLA-8-320.jpg)
![Test Adequacy Criteria
• Literature expresses as a form of measurable
code coverage [3]
• Status Quo Criteria:
– Statement coverage
• Most common adequacy criteria employed today
• Tools readily available to measure and report
• Extremely weak criteria
10](https://image.slidesharecdn.com/untitled-150223093544-conversion-gate02/85/The-Evolution-of-Testing-Methodology-at-AWS-From-Status-Quo-to-Formal-Methods-with-TLA-12-320.jpg)
![Test Adequacy Criteria
• Literature expresses as a form of measurable
code coverage[3]
• Perfect Criteria:
– Cover every execution path across every possible
state for the system
• States may be infinite
• Test space is exponential with path length
11](https://image.slidesharecdn.com/untitled-150223093544-conversion-gate02/85/The-Evolution-of-Testing-Methodology-at-AWS-From-Status-Quo-to-Formal-Methods-with-TLA-13-320.jpg)
![Testing Starts With Development
• Write tests and test support structure while working
on the implementation
– The spirit of “Test Driven Development” without
subscribing to the specific formula
• It is unclear how useful adhering to strict TDD concepts
really is [4]
• The emphasis it puts on test, and test thoroughness is
where the value is [5]
15](https://image.slidesharecdn.com/untitled-150223093544-conversion-gate02/85/The-Evolution-of-Testing-Methodology-at-AWS-From-Status-Quo-to-Formal-Methods-with-TLA-20-320.jpg)
![Anecdotes From QuickCheck Paper [6]
• Made them think harder about properties;
document the specification
• Need to think about the input domain to
exercise less probable paths
19](https://image.slidesharecdn.com/untitled-150223093544-conversion-gate02/85/The-Evolution-of-Testing-Methodology-at-AWS-From-Status-Quo-to-Formal-Methods-with-TLA-27-320.jpg)
![Anecdotes From QuickCheck Paper [6]
19](https://image.slidesharecdn.com/untitled-150223093544-conversion-gate02/85/The-Evolution-of-Testing-Methodology-at-AWS-From-Status-Quo-to-Formal-Methods-with-TLA-28-320.jpg)
![Anecdotes From QuickCheck Paper [6]
19](https://image.slidesharecdn.com/untitled-150223093544-conversion-gate02/85/The-Evolution-of-Testing-Methodology-at-AWS-From-Status-Quo-to-Formal-Methods-with-TLA-29-320.jpg)
![Formal Methods
• Precise specification of algorithms
• Tools to validate correctness
• We surveyed some of the systems and languages
for writing formal specifications, and ended up
finding what we were looking for in TLA+ [7]
• TLA+ gives us all possible executions over all
possible system states for algorithm designs
23](https://image.slidesharecdn.com/untitled-150223093544-conversion-gate02/85/The-Evolution-of-Testing-Methodology-at-AWS-From-Status-Quo-to-Formal-Methods-with-TLA-36-320.jpg)
![Real World Examples
• DynamoDB
– Replication protocols
– Membership handling
– Quorum Configuration Changes
• Other AWS projects [8]
– Low level distributed network protocol
– Internal distributed lock manager
– S3, EC2, EBS system management algorithms
26](https://image.slidesharecdn.com/untitled-150223093544-conversion-gate02/85/The-Evolution-of-Testing-Methodology-at-AWS-From-Status-Quo-to-Formal-Methods-with-TLA-42-320.jpg)
![28
[2] Hamilton, J. Challenges in Designing at Scale: Formal Methods in Building Robust
Distributed Systems. Perspectives Blog. July 2014;
http://perspectives.mvdirona.com/2014/07/03/ChallengesInDesigningAtScaleFormalMethodsInBuildingRobustDistributedSystems.aspx
[1] Barr, J. Amazon S3-The First Trillion Objects. Amazon Web Services Blog. June 2012;
https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/
[3] Zhu, H., et al. Software Unit Test Coverage and Adequacy. ACM Computing Surveys,
Vol. 29, No. 4, December 1997;
http://www.cs.toronto.edu/~chechik/courses07/csc410/p366-zhu.pdf
[4] Bulajic, A., Sambasivam, S., Stojic, R. Overview of the Test Driven Development
Research Projects and Experiments. Proceedings of Informing Science & IT Education
Conference (InSITE), 2012;
http://proceedings.informingscience.org/InSITE2012/InSITE12p165-187Bulajic0052.pdf
[5] Dalke, A. Problems with TDD. Dalke Scientific (dalkescientific.com), December 2009;
http://www.dalkescientific.com/writings/diary/archive/2009/12/29/problems_with_tdd.html](https://image.slidesharecdn.com/untitled-150223093544-conversion-gate02/85/The-Evolution-of-Testing-Methodology-at-AWS-From-Status-Quo-to-Formal-Methods-with-TLA-45-320.jpg)
![29
[6] Claessen, K., Hughes, J. QuickCheck: A Lightweight Tool for Random Testing of
Haskell Programs. ACM SIGPLAN Notices Volume 35 Issue 9, Sept. 2000;
http://www.eecs.northwestern.edu/~robby/courses/395-495-2009-fall/quick.pdf
[7] Newcombe, C., et al. Use of Formal Methods at Amazon Web Services. (pending ACM
publication), November 2013;
http://research.microsoft.com/en-us/um/people/lamport/tla/amazon.html
[8] Newcombe, C. Why Amazon Chose TLA+. Lecture Notes in Computer Science Volume
8477, June 2014; http://link.springer.com/chapter/10.1007%2F978-3-662-43652-3_3](https://image.slidesharecdn.com/untitled-150223093544-conversion-gate02/85/The-Evolution-of-Testing-Methodology-at-AWS-From-Status-Quo-to-Formal-Methods-with-TLA-46-320.jpg)
The Evolution of Testing Methodology at AWS: From Status Quo to Formal Methods with TLA+
- 1. The Evolution of Testing Methodology at AWS: From Status Quo To Formal Methods With TLA+ Tim Rath Principal Engineer AWS Database Services Amazon.com 1
- 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /aws-testing-tla
- 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
- 5. Services comprise large fleets of servers decomposed into smaller services 3
- 6. Many of which experience sustained exponential growth 4
- 7. S3 experienced exponential growth for 6 years to reach 1 trillion objects stored; less than a year later it reached 2 trillion objects [1] 5
- 8. DynamoDB processes millions of transactions per second in a single AWS region around the clock [2] 6
- 9. Systems and data are managed through subtle concurrent and distributed algorithms 7
- 10. “Must Haves” of Every Service 8 Security Durability Scalability Availability
- 11. General Test Strategy • Developers – Unit Tests – Integration Tests 9 •QA (release testing) •Functional Tests •Performance Tests •Stress Tests •Failure Tests
- 12. Test Adequacy Criteria • Literature expresses as a form of measurable code coverage [3] • Status Quo Criteria: – Statement coverage • Most common adequacy criteria employed today • Tools readily available to measure and report • Extremely weak criteria 10
- 13. Test Adequacy Criteria • Literature expresses as a form of measurable code coverage[3] • Perfect Criteria: – Cover every execution path across every possible state for the system • States may be infinite • Test space is exponential with path length 11
- 14. Test Adequacy Criteria • Real world practice further defines test adequacy criteria through an ad-hoc process: – Brain-storm test scenarios – Brain-storm stress test workloads – Brain-storm failure scenarios 12
- 15. Better testing of distributed algorithms • We look for strategies that: – Help to understand and protect algorithm invariants – Help expand test coverage – Allow thorough testing as early as possible in the development process 13
- 16. Development starts with specification 14
- 17. Development starts with specification 14
- 18. Development starts with specification 14
- 19. Development starts with specification 14
- 20. Testing Starts With Development • Write tests and test support structure while working on the implementation – The spirit of “Test Driven Development” without subscribing to the specific formula • It is unclear how useful adhering to strict TDD concepts really is [4] • The emphasis it puts on test, and test thoroughness is where the value is [5] 15
- 21. Testing Starts With Development 15
- 22. Assert system invariants • Asserts enforce the specification in code • Strong assert statements come from clear understanding of the specification 16
- 25. Generative Testing Formalization around randomized testing with invariant or “property” checking 17
- 26. Generative Testing 18 Test Case Generator Test Execution Validation Of Properties Against Result Properties Violated? Expected Properties Of Result Report Failure Case Loop No Yes
- 27. Anecdotes From QuickCheck Paper [6] • Made them think harder about properties; document the specification • Need to think about the input domain to exercise less probable paths 19
- 28. Anecdotes From QuickCheck Paper [6] 19
- 29. Anecdotes From QuickCheck Paper [6] 19
- 30. In Process Clusters 20 Process 1) Run multiple nodes in the same process as unit test 2) Insert arbitrary code into the communication channel 3) Ability to pipeline the inserted bits of code
- 31. In Process Clusters • Helps write better integration tests – Allows easy construction of intricate test scenarios – Integration testing of distributed components in a unit test environment • Helps write better stress tests – Direct control at the communication layer – Much faster, lower over-head test cycles 21
- 34. Informal Proofs • Requires deep thinking which promotes even better understanding of the algorithms • Hard to get right – can still lead to a false sense of security 22
- 36. Formal Methods • Precise specification of algorithms • Tools to validate correctness • We surveyed some of the systems and languages for writing formal specifications, and ended up finding what we were looking for in TLA+ [7] • TLA+ gives us all possible executions over all possible system states for algorithm designs 23
- 38. TLA+ • How the model works: – Init == <set of initial system states> – Next == <set of possible next actions> • That’s it! 25
- 39. TLA+ 25
- 40. TLA+ 25
- 41. TLA+ 25
- 42. Real World Examples • DynamoDB – Replication protocols – Membership handling – Quorum Configuration Changes • Other AWS projects [8] – Low level distributed network protocol – Internal distributed lock manager – S3, EC2, EBS system management algorithms 26
- 43. Thank You! 27
- 45. 28 [2] Hamilton, J. Challenges in Designing at Scale: Formal Methods in Building Robust Distributed Systems. Perspectives Blog. July 2014; http://perspectives.mvdirona.com/2014/07/03/ChallengesInDesigningAtScaleFormalMethodsInBuildingRobustDistributedSystems.aspx [1] Barr, J. Amazon S3-The First Trillion Objects. Amazon Web Services Blog. June 2012; https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ [3] Zhu, H., et al. Software Unit Test Coverage and Adequacy. ACM Computing Surveys, Vol. 29, No. 4, December 1997; http://www.cs.toronto.edu/~chechik/courses07/csc410/p366-zhu.pdf [4] Bulajic, A., Sambasivam, S., Stojic, R. Overview of the Test Driven Development Research Projects and Experiments. Proceedings of Informing Science & IT Education Conference (InSITE), 2012; http://proceedings.informingscience.org/InSITE2012/InSITE12p165-187Bulajic0052.pdf [5] Dalke, A. Problems with TDD. Dalke Scientific (dalkescientific.com), December 2009; http://www.dalkescientific.com/writings/diary/archive/2009/12/29/problems_with_tdd.html
- 46. 29 [6] Claessen, K., Hughes, J. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. ACM SIGPLAN Notices Volume 35 Issue 9, Sept. 2000; http://www.eecs.northwestern.edu/~robby/courses/395-495-2009-fall/quick.pdf [7] Newcombe, C., et al. Use of Formal Methods at Amazon Web Services. (pending ACM publication), November 2013; http://research.microsoft.com/en-us/um/people/lamport/tla/amazon.html [8] Newcombe, C. Why Amazon Chose TLA+. Lecture Notes in Computer Science Volume 8477, June 2014; http://link.springer.com/chapter/10.1007%2F978-3-662-43652-3_3
- 47. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/aws- testing-tla