Learn when and why to filter tests, and explore Android-specific test filters: @RequiresDevice, @SdkSuppress, @SmallTest, etc.
- [Voiceover] The Android JUnit Runner from the Android Testing Support Library, allows you to annotate your unit tests with indicators that suggest a number of circumstances about when your unit test should be executed or skipped. In other words, these annotations serve as execution filters that the runners use to determine whether or not to execute your unit test. Think of these annotations like JUnit's ignore annotation. But ignore with some condition that must be met before the test is ignored.
Currently Android and the Android JUnit Runner recognizes five of these filters. And I've list them in the table shown here. More on each of these in just a bit. These annotations can be placed on a unit test method or on the test class as a whole, thereby applying to all the test methods in the test class. The Requires Device annotation filter prohibits tests from being executed on an emulator. Some things can only be tested or tested effectively on a real device.
Check out this website for more information on emulator testing difficulties. Testing on a USB connection, making a phone call, or testing for blue tooth connectivity are just some of the limitations of using an emulator to test an app. When you have a test, specifically an instrumented test, that uses code that falls on an emulator limitation, you can mark a test with Requires Device annotation. Then the test is excluded from execution by Android JUnit Runner. Your app may be written to support multiple versions of Android.
Some parts of the app may only work when run on certain versions of the Android API. The SDK Suppress annotation allows you to specify the minimum Android API level on the device or emulator, before running the test. This annotation requires you specify a min SDK version integer when annotating a test as shown in my example here. The parameter is just an integer that is suggestive of the required Android API level, not the Android version number.
Here, I'm saying this test runs on the Android API level 23, which equates to Android version 6.0 or Marshmallow. For a list of the Android versions and API levels, see the Android Dashboard. If you are more familiar with the Android version codes like Gingerbread, Ice Cream Sandwich, and Jelly Bean, you can use an Android build version codes constant to specify the minimum API levels supported by the test. What is interesting, is that the parameter to the SDK Suppress, is just an integer, so you can actually suggest a future API level not yet released.
Like release 55, shown here. This could be useful if you think a piece of code or test is going to break with an upcoming release. The last set of filter annotations all help to classify how long a test should run, what resources it's using, and how frequently you expect to execute the test. This Google Test Blog does a great job of explaining when you want to annotate your test with either Small Test, Medium Test, or Large Test annotations. Generally speaking, when trying to determine what size filter to place on a test, look at the following factors.
One, how long will it take for the test to execute? In gross measures, tests taking milliseconds are considered Small Tests. Tests taking seconds are Medium Tests, and tests taking minutes are Large Tests. Two, what resources does the test utilize and need to complete its work? Small Tests use no resources, and don't require extra threads. Medium Tests may use file system, databases, threads, etcetera, but don't use network access or external systems.
Large Tests often use network access and external systems. For example, calling on a web service. Three, how often do you plan to execute the test? Small Tests are usually executed often, particularly when any piece of the code is changed or modified to check that nothing else has been upset by refactoring. Large Tests are usually left for the end of day system testing, Medium Tests are somewhere in between. Time now to try some of these filtering annotations.
I've started Android Studio and opened project 03_05_start. Here again, there is no end project for this video, as I won't be changing any code, just executing the existing code. In most cases, you don't change the way you execute your test when taking advantage of filters. The filters just inform the runner to execute or not execute the test based on the filters. In Main Activity Test I have one of each of the types of filters discussed in this video on each of the methods. To run my test I just right click on the test and run it as normal.
Tests are executed when they aren't filtered out. Since I'm running on an emulator and running at API level 23, two of my tests should be skipped. The Requires Device test and the Test Minimum SDK version, which is looking for a non-existent version 30. However, at the time of this recording, Android Studio 2.1 on the SDK have a bug, which is ignoring the Requires Device annotation. This will work on older versions of Android Studio and should be fixed with an upcoming release.
Unlike other filter annotations, or other unit testing work for that matter, in order to exercise unit tests with the Size filtering annotations, you must exercise your test using adb shell. This will not be used in Android Studio. Hopefully this will change and you can utilize these filters with tests in Android Studio soon, but as of this recording, it requires use of the adb shell. In order to exercise unit tests and include only the tests annotated with particular Size filter annotation, open a command prompt when in Windows or Terminal when in Linux or Mac environment, and change directories to the SDK platform tools folder within the Android installation directory.
In the terminal you'll enter the following command to exercise your test. The am instrument indicates I wanna run instrumented tests. The -w parameter instructs the shell to wait and keep the shell open until the instrumentation tests are finished. the -e option allows key value pairs to be provided in the request. In this case, the key value pairs allow the Size parameter to be specified. Using Size and a Small, Medium, or Large value, representing Small Test, Medium Test, and Large Test, I specify which test methods I want executed.
To complete the request to adb shell, I specify the test package and the JUnit Runner I want to execute my test, separated by a slash character. The package name is typically the name of the application package as specified in the Android manifest xml file followed by dot test. The runner I want to specify is Android Support Test Runner Android JUnit Runner. Returning to Main Activity Test, and for example sake, I've marked one of my tests with Large Test, that is the testLargeTest method.
Now, in the adb shell I'm going to ask the shell to execute Large Test. Note the Size Large key value pair in my command request. And we see the result of executing one test. To demonstrate what happens if I provide a Size filter and not matching any Size annotations, I'll change my request in the adb shell to execute Medium Tests, without modifying any of the code. And now we see zero tests were executed.
- Why unit test?
- Unit testing tools
- Local testing with JUnit
- Working with JUnit assertions, rules, and categories
- Executing instrumented tests
- Filtering tests
- Testing with UI Automator and other testing tools