Multiplication and division of calabash tests
2 likes•441 views
Rajdeep Varma discusses parallel_calabash, a tool they created to run mobile test automation in parallel. It addresses long test run times by distributing Cucumber feature files across multiple device instances and processes. This allows taking advantage of modern multi-core machines and significantly increases testing velocity while maintaining test coverage. It can run tests on multiple real or emulator devices simultaneously and generates separate JUnit-style reports for each device. With parallel_calabash, they achieved a 70% increase in testing speed while gaining the ability to test on multiple device configurations in parallel.
![How it works?
<process spawner>
1_digit_addition
2_digit_addition
3_digit_addition
4_digit_addition
5_digit_addition
6_digit_addition
7_digit_addition
8_digit_addition
ENV[ADB_DEVICE_ARG] = emulator-5554
ENV[TEST_PROCESS_NUMBER] = 1
Process
Spawner
ENV[ADB_DEVICE_ARG] = emulator-5556
ENV[TEST_PROCESS_NUMBER] = 2
ENV[ADB_DEVICE_ARG] = emulator-5558
ENV[TEST_PROCESS_NUMBER] = 3
ENV[ADB_DEVICE_ARG] = emulator-5560
ENV[TEST_PROCESS_NUMBER] = 4
S
U
M
M
A
R
I
Z
E
R
E
S
U
L
T
stdout](https://image.slidesharecdn.com/multiplicationanddivisionofcalabashtests-150608144950-lva1-app6891/85/Multiplication-and-division-of-calabash-tests-21-320.jpg)
![Reporting
config/cucumber.yml
default: --format html --out my_report_<%=ENV['DEVICE_INFO']%>.html
Reports from 3 devices:
my_report_GT_N7100.html
my_report_Samsung_Galaxy_Note_2___4_3___API_18___720x1280_1.html
my_report_Samsung_Galaxy_Note_2___4_3___API_18___720x1280_2.html](https://image.slidesharecdn.com/multiplicationanddivisionofcalabashtests-150608144950-lva1-app6891/85/Multiplication-and-division-of-calabash-tests-22-320.jpg)
![what did we achieve?
• 70% increase in testing velocity.
• Scalability
• Reporting
• Process specific action using
ENV[TEST_PROCESS_NUMBER]](https://image.slidesharecdn.com/multiplicationanddivisionofcalabashtests-150608144950-lva1-app6891/85/Multiplication-and-division-of-calabash-tests-27-320.jpg)
Multiplication and division of calabash tests
- 2. About Myself • Rajdeep Varma • 5 years in Quality Engineering. I work at ThoughtWorks • Passionate about test automation and agile practices • Tested a number of web apps and recently started testing mobile apps
- 3. I will talk about • Our Problem: A. Long running automation B. Running Automation on Multiple devices • Our Solution: parallel_calabash
- 4. Typical CI Pipeline Of Mobile Project Git Push Compile + Unit Test Build App UI Tests Distribute “Fast Feedback”
- 6. Stats • 5 git repo • Avg. 30 commits a day • Approx. 100 Automated UI tests. • “GitPush” TO “TestedOK” > 1 Hrs
- 7. With slow tests Git Push UI Tests Deploy to next stage Umm…slow tests—Chai Break—damn! test failed :( Fixed! <pain>its so slow<pain/>— Pray —
- 8. LOL
- 9. :(
- 10. Patience Is Not Always A Virtue
- 11. Choose One a) Lots of automated tests for good coverage b) Faster automation run c) BOTH!
- 13. Our tool-set • MobileWeb: Cucumber + waitr-webdriver + parallel_tests • AndroidApp: Calabash
- 14. Options • Multiple parallel builds in CI • Proprietary cloud based tools • Create a tool that matches our need http://images.clipartpanda.com/choice-clipart-fork-choice-hi.png
- 16. Demo
- 21. How it works? <process spawner> 1_digit_addition 2_digit_addition 3_digit_addition 4_digit_addition 5_digit_addition 6_digit_addition 7_digit_addition 8_digit_addition ENV[ADB_DEVICE_ARG] = emulator-5554 ENV[TEST_PROCESS_NUMBER] = 1 Process Spawner ENV[ADB_DEVICE_ARG] = emulator-5556 ENV[TEST_PROCESS_NUMBER] = 2 ENV[ADB_DEVICE_ARG] = emulator-5558 ENV[TEST_PROCESS_NUMBER] = 3 ENV[ADB_DEVICE_ARG] = emulator-5560 ENV[TEST_PROCESS_NUMBER] = 4 S U M M A R I Z E R E S U L T stdout
- 22. Reporting config/cucumber.yml default: --format html --out my_report_<%=ENV['DEVICE_INFO']%>.html Reports from 3 devices: my_report_GT_N7100.html my_report_Samsung_Galaxy_Note_2___4_3___API_18___720x1280_1.html my_report_Samsung_Galaxy_Note_2___4_3___API_18___720x1280_2.html
- 23. Optimization in test distribution • Issue with distribution based on cucumber “featue” files • Distribution based on “Scenario” instead of features • --group-by-scenarios
- 24. Advantages of fast tests • Fast Feedback • Faster development cycle for BDD environment • Modern machines have multiple cores and parallelization can make Maximum use of Hardware • Faster tests makes developers & Testers happy
- 25. I will talk about • Our Problem: A. Long running automation B. Running Automation on Multiple devices • Our Solution: parallel_calabash
- 26. Device Fragmentation parallel_calabash -a <path/to/apk> features --concurrent
- 27. what did we achieve? • 70% increase in testing velocity. • Scalability • Reporting • Process specific action using ENV[TEST_PROCESS_NUMBER]
- 29. Choose One a) Lots of automated tests for good coverage b) Faster automation run c) BOTH!
- 30. TIME
- 32. References: • https://github.com/grosser/parallel_tests • http://martinfowler.com/ • http://continuousdelivery.com/ • http://watirmelon.com/
- 33. ?