Instrumentation 101

Editor's Notes

• #2: Hello, My Name is Alex Bannerjee and presenting with me is Daniel Joyce.We are software engineers at Apkudo.Today, we will be giving a brief overview on Android Instrumentation for beginner app developers---------------------------------------Standard IntroductionMaybe mention what we do at Apkudo.Presentation is focusing on application instrumentation for android developers
• #3: Our goal today is to talk about android instrumentation tools and their roles in the app development processSo first and foremost, we ask Why Bother? No one likes testing. It’s boring and tedious and time consuming. However, Its role is very important to the process. In the long run, it will save you time hunting down bugs save you money because you’ll be able to launch your apps on schedule facilitates team development since each member will have confidence that their work is bug free Fortunately, android supports many tools for testing most are easy to use and pick up there is a variety of tools available, each fulfilling a different role in the testing ecosystemWhat tools are available? Today we will be showcasing four popular tools: Monkey, MonkeyRunner, Robotium, UIAutomator We’ll show you how to use each And there are plenty of resources available online for each toolWhat’s Next? Finally, we’ll wrap up and talk about the next steps in testing-------------------------------------------In this presentation we’llbeaddressingsome questions frequentlyasked by appdeveloperswhenitcomes to testingOur goal is to answer these common questions asked App DevelopersWhy Bother? No one likes testing. It’s boring, tedious, etc. Its important, in the long run it will save time, money, save you launch headaches. Good thing, android supports many tools for testing. Most are easy to use, and support scripting/automating your tests. You’re all programmers, so just do it.What tools are available? Showcase several [popular | common] tools: Monkey, MonkeyRunner, UI Automator, Robotium. Resources for each tool is widely available on google’s android developer siteHow do I use them? Step by Step guide on how to get set up and begin using each tool. Demo different Methods of testing (starting off with simple single commands to scripting test suites.) Look at understanding the test results such as Monkey Logs and Logcat.What’s next? Automating tests for regression testing (test & analysis suite)
• #4: Monkey is a simple and easy to use command line tool that is part of the Android SDK.Monkeys only purpose is stress testing your application. This means, testing the things you would normally not test or think to test. Monkey provides a stream of random input events while your application is running with the primary goal of determining how your app preforms under such a load. This will test all kinds of strange input events, and I assure you, that your app will crash at some point, under a corner case that you never imagined.Monkey is not for testing specific app features and it is not smart. Smart being, scriptable, (you don’t tell monkey what to do.) Finally, monkey is not a “cure-all”. It wont find the bugs that other types of testing will find. Monkey is simple, easy, and not terribly exciting, but its most definitely useful.----------------------------------Monkey is a command line tool that exists on android devices.You can find the monkey binary in /system/bin on your deviceYou can run monkey from your [base | development] machine by calling “adb shell monkey”Monkey is: For Stress Testing For Testing things that you thought could never happen such as: Weird UI sequences: press button 1 before button 2 Unexpected inputs: turning off wifi, keyboard inputs during toast press menu button before UI loads even if button is not visible on the screen Race conditions: press UI elements really quicklyMonkey won’t: Test specific features of your application. Monkey is meant to be random, if you find yourself writing specific inputs using monkey, you should consider using another tool (MonkeyRunner)
• #5: To use monkey, you simply enter one command in the terminal (with your device connected of course):adb shell monkey –p <package-name> <count>You can also use other command line flags, such as: -v for extra verbosity in the monkey output -s to change the random seed used for input events -pct-<input-type> for specify the percentage of each type of input event -throttle the delay between touch eventsIf you decide to use the v flag you’ll get output of the following form:Header, shows some of your command line options. Event percentages which show how often a specific type of key input will be injected by monkey. Sending events, shows each event and the specific value sent to the device Footer, Total number of events injected, network stats, total number of dropped events Errors (and debug info if you have that option enabled)You should also run logcat while running monkey to get an idea of your apps performance.Bottom line: use monkey, its too easy not to use.-----------------------------------Usage: monkey runs in adb shell monkey executible exists on the device in /system/bin no need to modify your apk to test using monkey if you don’t know your package name you can do:aapt dump badging com.yourapp.apkOutput: If you use the verbose option (-v) you will get the following output Header, shows some of your command line options. Event percentages which show how often a specific type of key input will be injected by monkey. Sending events, shows each event and the specific value sent to the device Footer, Total number of events injected, network stats, total number of dropped events Errors (and debug info if you have that option enabled)Bottom Line: Use Monkey if you’re not going to test with any other tool Quick and simple, easily scriptable in bashADDITIONAL NOTES:How Monkey Works:Source : platform/development/cmds/monkeyProcesses Command Line optionsLoads the Package ListConnects to System InterfacesActivity ManagerPackage ManagerWindow Manager Generates a list of activities monkey can access given categories and package restrictions(MainApps)Determines which event source to use (Random, Network, or Script)Validate Event Source (check header)Check if we’re profilingRun Monkey by calling runMonkeyCycles (loop through all MonkeyEvents)Report StatsImport android.hardware.input.InputManagerEvents are injected via MonkeyEvent’sinjectEvent(WindowsManager, ActivityManager, int verbose)which calls InputManager.getInstance().injectInputEvent(InputEvent, int mode)IInputManager.aidlWhen using monkey, you have the option of having 3 different sources for your keyeventsBy Default, if port or script isn’t specified Running Monkey With Script File:Command Line: Checks for script file name (-f option) Optional: Checks for setup filemSetupFileNamemScriptFileNamesMonkeySourceScript.javagetNextEventreadNextBatchreadLinesprocessLinehandleEventIn HandleEvents, lines should be in the format @code cmd(arg1,arg2,arg3….)LaunchActivity(String pkg_name, String cl_name)DispatchKey(long downtime, long eventTime, int action, int code, int repeat, intmetaState, int device, intscancode)
• #6: In addition to stress testing, you may want to test a specific sequence of inputsIn which case, you can use MonkeyRunnerMonkeyRunner is an android sdk tool located in sdk/tools/MonkeyRunner is a smarter monkey in that: It allows for a better structured test. You can do Functional Testing: Set of tests that makes sure your specific feature does what it’s supposed to Regression Testing: Set of tests that you would run every time you make a changeMonkeyRunner allows you to test your application from start to finish using scripts and it allows you to test your application’s stability by comparing the outputs to a set of outputs known to be correct A nice feature of MonkeyRunner is it supports scripts written in Python Python is a nice language which is easy to use and easy to pick up, but hard to master In addition, MonkeyRunner supports capturing device screenshots Useful for knowing if your UI is being displayed correctly (i.e. correct labels on buttons) in an automated mannerFrom our experience using MonkeyRunner, we’ve noticed it to be not that fast in executing touches on the device compared to monkeyAll in all, MonkeyRunner seems to be best suited for regression testing and integration testing----------------------------------------------------------Android sdk tool found in sdk/tools/MonkeyRunner is: A Smarter Monkey For the following: Functional Testing : Automated start to finish test of an application Regression Testing : Test application stability by comparing its output to a values known to be correct Supports writing test scripts in python Allows you to extend MonkeyRunner API with plugins Allows you to take a screenshot of the device Useful for knowing if your UI is being displayed correctlyMonkeyRunner isn’t: Fast, touches are instrumented slower than on monkeyTesting new scripts is fast using MonkeyRunner (no need to rebuild java files, push jars to the device)Writing scripts is python is quick and easy
• #7: Step 1: You should learn Python Know that MonkeyRunner uses Jython Meaning that your python scripts can interface with java classes Overall this means your test scripts will be written in Python opposed to in Java (as we’ll see for some other tools)Step 2: Understand how to use the three main modules of MonkeyRunner:MonkeyRunner – Connect to deviceMonkeyDevice – Call functions on the device (examples below)MonkeyImage – Take screenshots The MonkeyRunner module allows your to connect to your device Returns a device object (MonkeyDevice) That device object allows you to call various methods such as:installPackage touch (pass in touch type)takeSnapshotstartActivity Use MonkeyImage to capture screenshots of your device You can use method sameAs to get a diff of your image with an image you are expecting for regression testingStep 3: run your script
• #8: Robotium is a powerful tool and API for testing your Android Applications.Robotium is more in depth and more robust than the tools we’ve seen so far. The basic idea of robotium, is to create a testing APK that will exercise your target app through test cases that you write. In that way, it is very much like any standard testing framework (aka. Python’s unit test etc.) Robotium’s tag line is, “Like Selenium for Android”, so, if you come from a web development background, Robotium should be familiar ground for you. One more thing to note, Robotium is very fast, which is a great boon to the testing experience. On the downside,Robotuim requires some boilerplate. You’re going to have to create an APK, build with the robotium JAR, and in general, be aware of the internals of the app you wish to test. That being said, it is quite powerful and, some might say, the de-facto testing tool for android apps.-------------------------------------Robotium is Is written in Java and utilizes Android’s Unit Testing Framework which uses JUnit You can make tests generic (one test apk to test a range of applications rather than writing one test per application) Requires you to build test apk’ and then test it through adb Include robotium in your build path when you build your apk Write test projects into your apk Solo is robotium’s test Class Gives you more insight on the UI get text boxes, image buttons, radio buttons, toggle buttons, scroll views, etc. method to assert the amount of available memory in the system Essentially, you are writing a test apk to test your main apk Like Selenium if you are a web developer and are familiar with that sort of thing Execution of tests is fast Robotium isn’t: Quick and easy to set up, you have to write a test apk Sit down and write a bunch of test cases
• #9: Robotium requires the robotium JAR which can easily be downloaded from the Robotium website. You’re going to need to know your APK’s package name and signature, though, you should have both if you are developing the app you wish to test. Tests are set up by extending the main test class, and writing methods for each test case.The Robotium Solo class provides the interface for testing on the device. That is, all of your testing commands will be issued from a solo object. As is the norm with testing, you’ll want to use assert to ensure test results. If you take a look at the API later, you’ll see that Robotium provides a bunch of methods, allowing you to do a lot with the app (switching activities being quite useful.)Finally, Robotium can be run from eclipse or the command line (your preference) and provides output regarding test case success, failure, and error.-------------------------------------Note: Much easier running a robotium test through Eclipse than through command lineStep 1: AddRobotium’s jar into your build path for your applicationStep 2:Robotium’s Solo class gives you access to your applications UI Follows a standard testing framework perform actions assert results run next testStep 3: Run it in eclipse by going to “Run application as JUnit test” or Run it in command line as:adb shell am instrument -w com.jayway.test/android.test.InstrumentationTestRunner
• #10: UIAutomator is a testing framework that has been added fairly recentlyIt is Part of the Android SDK revision 21 and upAnd it requires your test device to have android API version 16 or higherUIAutomator consists of: a command-line tool found on the device a Java library (uiautomator.jar found in the sdk/tools/) which you use in your tests Android SDK tool uiautomator viewer which comes in handy for writing your tests (we will see this in action in the demo)Features of UIAutomator: Capable of referencing UI Objects by name (opposed to clicking on them via coordinates using other testing tools) (we will see how to reference these UI Objects using the UI Selector class and uiautomator-viewer) Some methods you can call in UIObjectsisEnabledisFocusedisScrollable Packaged with Uiautomator viewer which Captures screenshot of the device And Lists all the UI objects (great for identifying which components you want to test)UIAutomator isn’t: Capable of recognizing OpenGL UI components (such as start button in Angry Birds)-----------------------------------------------Comes with Android SDK revision 21+ and requires android API version 16+UIAutomator is: Capable of referencing UI objects by name (instead of having to click on them via coordinates and touch events)UISelector to search for components with a specific label You can use that UISelector to create a UIObject which you can then call the click() or get various other info on that component (isEnabled, isChecked, isFocused, isScrollable, etc.) Packaged with UI Automator Viewer which: Captures screenshot of the device And Lists all the UI objects (great for identifying which components you want to test)UIAutomator isn’t: Capable of recognizing OpenGL UI components (such as start button in Angry Birds)
• #11: Step 1: Create a Java Project Demo how to create a project in the terminal (not using Eclipse) Steps to building Test: Export Environment Variable export ANDROID_HOME=/home/alexbannerjee/Dev/sdk/toolsbuild.xml needs ANDROID_HOME to build Create Ant Build Files android create uitest-project -n Test -t "android-16" -p /home/alexbannerjee/workspace/Test Ant Build Push compiled JAR to the phone (i.e. /data/local/tmp/) Run Testadb shell uiautomatorruntest /data/local/tmp/YourJar.jar -c yourpackage.YourTest
• #12: So far, the tools we’ve seen, with the exception of Robotium, have done black box testing of the UI. However, white box testing, whereby you test units internally, is just as important. There are a whole host of other tools that fill this role. So, today you’ve received a primer on the types of tools available, what they are capable of, and the roles that they fill. We hope we’ve provided a good launching point for you to get started. The biggest thing to take away from this presentation is that you should make it a habit of using some combination of these tools to test your apps. Develop a testing framework that you use and stick to it. You are all programmers, so it should be no trouble for you to automate your tests. Don’t forget to test before every pull request!