Cross mobile testautomation mit Xamarin & SpecFlow
7 likes3,979 views
This document summarizes a presentation on cross-platform mobile test automation using Xamarin and SpecFlow. It discusses using Xamarin to build native mobile apps with shared C# code, SpecFlow for writing acceptance criteria in Gherkin, and testing the user interface of mobile apps using Appium. It also describes testing at the controller layer for faster test execution. Automating tests is recommended to improve quality and provide faster feedback, but manual testing still has value for exploring apps and hard-to-automate checks.
![11
Xamarin.Android
twitterListView.setOnItemClickListener (new OnItemClickListener () {
public void onItemClick (AdapterView<?> parent, View view, int position, long id) {
Intent tweetDetails = new Intent (TwitterScreen.this, TweetDetailsScreen.class);
tweetDetails.putExtra ("TweetID", TwitterFeed[position].ID);
startActivity (tweetDetails);
}
};
twitterListView.ItemClick += (s, e) => {
var tweetDetails = new Intent (this, typeof (TweetDetailsScreen));
tweetDetails.PutExtra ("TweetID", TwitterFeed[e.Position].ID);
StartActivity (tweetDetails);
};
C#](https://image.slidesharecdn.com/fpwfwbltqk2zzvawapvs-signature-a7a45fb26d0c2689625accc93815fab6b274c891bc12d89d27ed12ec47daceeb-poli-150424023355-conversion-gate01/85/Cross-mobile-testautomation-mit-Xamarin-SpecFlow-11-320.jpg)
Cross mobile testautomation mit Xamarin & SpecFlow
- 1. COPYRIGHT, TECHTALK - WWW.TECHTALK.AT ANDREAS WILLICH (@SABOTAGEANDI) CHRISTIAN HASSA (@CHRISHASSA) Mobile Quality Crew Meetup, Vienna, April 23rd 2015 Cross Mobile Testautomation with Xamarin & SpecFlow
- 2. 2 Topics • Xamarin • Test automation in agile projects • Specification-By-Example • SpecFlow • UI automation on mobile devices • Mobile app automation with SpecFlow
- 3. 3 Xamarin
- 4. 4 Native development for each platform
- 7. 7 Compilation and run time Android APK Android/Linux kernel IL code for platform specific profile VS solution with projects iOS APP Android APK Shared Code (C#,VB.NET) Mono CLR for iOS (Monotouch) Mono CLR for Android (Monodroid) Multi-Platform profile assembly Dalvik Android.* Java.* IL+JIT AOT ARM Binary iOS App (C#) Android App (C#) Xamarin.iOS profile assembly Xamarin.Android profile assembly iOS/Cocoa Touch iOS APP ARM binary
- 8. 8 Native mobile apps with C#/Xamarin 1. Reuse existing .NET/C# Know-How 2. Powerful platform and tools (Linq, Xml, Events/Delegates, Parallel Programming, Visual Studio/TFS, …) 3. Shared code for multi-platform apps
- 11. 11 Xamarin.Android twitterListView.setOnItemClickListener (new OnItemClickListener () { public void onItemClick (AdapterView<?> parent, View view, int position, long id) { Intent tweetDetails = new Intent (TwitterScreen.this, TweetDetailsScreen.class); tweetDetails.putExtra ("TweetID", TwitterFeed[position].ID); startActivity (tweetDetails); } }; twitterListView.ItemClick += (s, e) => { var tweetDetails = new Intent (this, typeof (TweetDetailsScreen)); tweetDetails.PutExtra ("TweetID", TwitterFeed[e.Position].ID); StartActivity (tweetDetails); }; C#
- 12. 12 Development environment OS Mac OSX, Windows Windows UI Designer Android, iOS Build Android, iOS (via Xamarin Build Host on Mac OSX) Debug Android (Device and ARM/x86 emulator) iOS (Device and emulator on Mac)
- 13. 13 .NET libaries availabile on Xamarin Assembly feature set mscorlib.dll SL OpenTK.dll OpenGL/OpenAL System.dll SL+ System.Core.dll SL System.Data.dll NET35- System.Data.Service.Client.dll vollständiger OData Client System.Json.dll SL System.Runtime.Serialization.dll SL System.ServiceModel.dll SL (WCF) System.ServiceModel.Web.dll SL+ System.Transactions.dll NET35 System.Web.Services NET35- System.Xml.dll NET35 System.Xml.Linq.dll NET35 Mono.CompilerService s.SymbolWriter.dll Für Compiler Mono.Data.Sqlite.dll ADO.NET für SQLite Mono.Data.Tds.dll TDS Protokoll Support Mono.Security.dll Cryptographic API monotouch.dll iOS Bindings Mono.Android.dll Android Bindings
- 14. 14 Xamarin.Forms One UI definition for 3 platforms (iOS, Android, WP) in XAML (data binding) translates to native UI controls on each platform can be mixed with custom UI
- 16. 16 Test automation becomes expensive when … • automating manual tests • automation makes tests unreadable • automating only after implementation structure readability right moment
- 17. 17 Structure Manual tests Asserts Multiple combined features Structure ACT-ASSERT- ACT-ASSERT- ACT-ASSERT- … Dependent features Long test path with high chance to break Cause and impact of error hard to trace Automated Check Single aspect of a single feature ARRANGE – ACT – ASSERT Independent features Short test path with lower chance to break Cause and impact of error easy to relate
- 18. 18 What to automate User journeys Acceptance- criteria Units exploratory testing Test automation pyramid Source: Mike Cohn many few hard easy Automatability
- 19. 19 // Go to web page 'http://localhost:40001/' using new browser instance BrowserWindow localhostBrowser = BrowserWindow.Launch( new System.Uri(this.RecordedMethod1Params.Url)); // Click 'Register found item' link Mouse.Click(uIFundstückerfassenHyperlink, new Point(56, 9)); // Click 'Save' button Mouse.Click(uISpeichernButton, new Point(44, 14)); int fundNr1 = int.Parse(uIFundNr127Pane.InnerText.Substring(9)); // Click 'Register found item' link Mouse.Click(uIFundstückerfassenHyperlink, new Point(63, 7)); // Click 'Save' button Mouse.Click(uISpeichernButton, new Point(34, 11)); int fundNr2 = int.Parse(uIFundNr128Pane.InnerText.Substring(9)); Assert.IsTrue(fundNr1 + 1 == fundNr2); // Click 'Close' button Mouse.Click(uICloseButton, new Point(26, 11)); Readability
- 20. 20 A readable test case Scenario: New found items should receive a consecutive number for the current year Given the previous found item of the current year had the number 145 When I register a new found item Then the last found item of the current year should have the number 146
- 21. 21 When to test (right moment) Acceptance criteria checks (ATDD, BDD) Unit Tests (TDD) business view technical view Exploratory tests User Journey tests Performance, Scalability, Usability,Security, … productdefinition productcritique New dimension: defining the product Synergy: Specification of requirements and tests Agile Testing Quadrants: Brian Marick
- 22. 22 Automated checks don’t replace manual testing User journeys Acceptance- criteria Units exploratory testing manual check in definition of done for stories Main success paths Undiscovered acceptance criteria No/(fewer) manual regression checks Fewer paths are enough More time for exploration
- 24. 24 Agile “requirements” Specification- By-Example Image from: 50 Quick ideas to improve your user stories: https://leanpub.com/50quickideas Why? Outcome How? Specification
- 25. 25 Agile specifications define upfront define just-in-time Code Reminder for a conversation Goals Impacts Epics User Stories Acceptance criteria Why? Outcome How? Specification
- 26. 26 public void CalculateDiscount(Order order) { if (order.Customer.IsNew) order.FinalAmount = Math.Round(order.Total * 9/10); } Register as “bart_bookworm” Go to “/catalog/search” Enter “ISBN-0955683610” Click “Search” Click “Add to Cart” Click “View Cart” Verify “Subtotal” is “$33.75” We’d like to encourage new customers to buy in our shop. Therefore we want offer 10% discount for their first order. Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com
- 27. 27 Acceptance criteria Agile specifications define upfront define just-in-time Code Goals Impacts Epics User Stories Bugs Why? Outcome How? Specification
- 28. 28 Specification with examples Examples … make abstract descriptions better understandable are usually not formally documented or exchanged Examples Tests Requirements describe validate fulfillment of consist of
- 29. 29 Register as “bart_bookworm” Go to “/catalog/search” Enter “ISBN-0955683610” Click “Search” Click “Add to Cart” Click “View Cart” Verify “Subtotal” is “$33.75” public void CalculateDiscount(Order order) { if (order.Customer.IsNew) order.FinalAmount = Math.Round(order.Total * 9/10); } We’d like to encourage new customers to buy in our shop. Therefore we want offer 10% discount for their first order. Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com
- 30. 30 Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com New customers should receive 10% discount: Given the customer hasn‘t ordered yet When the customer adds a book for EUR 37.50 to the shopping cart Then the shopping cart shows a total price of EUR 33.75.
- 31. 31 Actually, this is not quite right: Books on sale should be excluded! Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com
- 32. 32 “Happy Path” Technical feasibility Exceptions, border cases 3 Amigos Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com
- 33. 33 Collecting Acceptance Criteria “I would try to put a book into the shopping cart …” “I would try to remove a book from the shopping cart…” “I’d check whether the shopping cart is empty, when I enter the shop …” Books can be added to shopping cart. Books can be removed from shopping cart. Shopping cart should be empty when entering the shop. ... ? … As a potential customer I want to collect books in a shopping cart So that I can order several books at once. “Imagine this story is already implemented: How would you verify it?” “I would try to add 1000 books to the shopping cart …”
- 34. 34 Exploration through examples Books in catalogue: Title Author Specification-By-Example Gojko Adzic Impact Mapping Gojko Adzic Explore It! Elisabeth Hendrickson Competitive Engineering Tom Gilb … I want to find books in the catalogue by author and title Search for … Books found … Spec Specification-By-Example Hend Explore It! et Explore It!, Competitive Engineering Context Action Assertion
- 35. 35 Key examples: Breaking the model Books in catalogue: Title Author Specification-By-Example Gojko Adzic Impact Mapping Gojko Adzic Explore It! Elisabeth Hendrickson Competitive Engineering Tom Gilb … I want to find books in the catalogue by author and title Search for … Books found … Spec Specification-By-Example Hend Explore It! et Explore It!, Competitive Engineering What happens, if I search for “Explore Specification”? Can I search for single characters, e.g. “e”? Is the number of search results limited, or paged? Is the search also performed in the sub-title of a book?
- 36. 36 Purpose of the examples • Shared understanding: acceptance criteria • Documentation: specification details • Regression-tests: violated specifications
- 37. 37 SpecFlow
- 38. 38 SpecFlow – BDD for .NET #62 in Visual Studio Gallery based on popularity (Apr 23 2015) #303 most downloaded package on NuGet in the last 6 weeks (Apr 23 2015) ~1‘400 unique daily visitors on specflow.org > 30’000 active users > 40 contributors
- 39. 39 Feature: Description of feature or user story Szenariogrundriss: Beschreibung des Akzeptanzkriteriums Szenario: Beschreibung des Akzeptanzkriteriums Gherkin Feature Files Background: context for all scenarios in the feature file Scenario: Description of acceptance criterion Angenommen/Wenn/Dann: Automatisierte Szenario Schritte Given/When/Then: automated scenario steps Scenario Outline: Description of acceptance criterion Angenommen/Wenn/Dann: Automatisierte Szenario Schritte Given/When/Then: automated scenario steps with <place holder> Examples: table with examples for <place holders>Examples: table with examples for <place holders> Given: automated scenario steps @tagname @tagname @tagname @tagname
- 40. 40 Abstract acceptance criteria As a shop visitor I want to collect books in my shopping basket so that I can purchase multiple books at once. Books can be added to the shopping basket Books can be removed from the shopping basket Shopping basket is initially empty The same book can be added multiple times to the shopping basket
- 41. 41 Scenarios: Examples in Gherkin As a shop visitor I want to collect books in my shopping basket so that I can purchase multiple books at once. Books can be added to the shopping basket Given my shopping basket is empty When I add the book “Harry Potter” to my shopping basket Then my shopping basket should contain 1 copy of “Harry Potter”
- 42. 42 As a shop visitor I want to collect books in my shopping basket so that I can purchase multiple books at once. Books can be added to the shopping basket Scenarios: Examples in Gherkin Given my shopping basket contains 1 copy of “Harry Potter” When I add the book “Harry Potter” to my shopping basket Then my shopping basket should contain 2 copies of “Harry Potter” The same book can be added multiple times to the shopping basket
- 43. 43 The same book can be added multiple times to the shopping basket Structure of Scenarios Given my shopping basket contains 1 copy of “Harry Potter” When I add the book “Harry Potter” to my shopping basket Then my shopping basket should contain 2 copies of “Harry Potter” Title: Describes intention/abstract acceptance criterion Arrange: Context, describes state of the system Act: Execution of the feature Assert: Assertion of observable behaviour And I should see the warning: “Book already existed in basket” Triple-A constraint “Checks” Chaining up steps
- 44. 44 Automated continuous validation Given my shopping basket contains 1 copy of “Harry Potter” When I add the book “Harry Potter” to my shopping basket Then my shopping basket should contain 2 copies of “Harry Potter” System „Step Definitions“ are binding individual steps to an automatable interface of the application. Automatable interface UI Automation Automation does not necessarily have to bind to the UI. Automatability of system is supported/evolving with development.
- 45. 45 UI automation on mobile devices
- 46. 46 Available tools/libraries • Android • Android 2.3+: Instrumentation • Android 4.2+: UiAutomator • selendroid • Appium • Robotium • Calaba.sh • Xamarin.UITest • iOS • UIAutomation • Appium • ios-driver • Calaba.sh • Xamarin.UITest • Frank • KIF (Keep it functional)
- 47. 47 Selenium: cross platform, existing know how • Android • selendroid • Appium • iOS • Appium • ios-driver
- 48. 48 Appium (www.appium.io) • Provides WebDriver API (JSON Wire Protocol) • Uses native UI Automation library • Can test native & hybrid Apps, mobile web apps • Clients for C#, Java, Ruby, Python, PHP, JavaScript • Works with devices and emulators
- 49. 49 Examples
- 50. 50 Automation with SpecFlow and Appium
- 51. 51 SpecFlow & Appium • Write acceptance criteria independent of the platform • Write bindings against a Selenium- like- API • Run it on a devices or emulator
- 52. 52 Demo
- 53. 53 SpecFlow automation on the controller layer
- 54. 54 Different levels of automation Controller Business Layer Data Layer Model View Device automation Trigger behaviour through controller Assert behaviour on model, db, .. Setup pre-conditions through service interfaces Out-of-process In-process
- 55. 55 How? • Reuse the same feature files • Write new bindings against the Controller/ViewModel/…
- 56. 56 Demo
- 57. 57 Faster feedback from tests UI Automation Test Controller Test
- 58. 58 Why is it faster? • No overhead for deploying and starting the app • No need to wait for UI animations to finish • No delays for searching UI controls
- 59. 59 Advantages Controller Testing • Faster • Better testable App architecture (Xamarin.Forms & MVVM)
- 60. 60 Controller tests cannot fully replace UI tests • UI tests are still needed • Diversity of mobile devices • Changes of manufacturer • Different Hardware Screen sizes Densities … • Automated UI tests are still faster than manual testing
- 61. 61 Conclusions
- 62. 62 Excuses for avoiding mobile test automation • “This is just a small app” • “Only complex end-end tests would make sense” • “Maintaining automated tests is too costly” • Diversity of devices • Small apps also run into regressions • Bad feedback on app store cannot be undone
- 63. 63 Automated testing in mobile App development • Business readable automated checks • specification of acceptance criteria • test first development • Testability is an ongoing architectural concern • every new feature needs to be testable • not every acceptance criterion needs to be tested through UI • Automated tests don’t replace manual testing entirely • exploratory testing • assertions that are hard to automate • …
- 64. 64 Questions and further information Andreas Willich ([email protected]) Christian Hassa ([email protected])