There is documentation on the Android Developer website sharing tips to run Android UI tests via the command-line which I’ve found super useful. It provides examples on how to run specific tests (instead of running them all) including running by package, class or method. This documentation is fantastic, but it doesn’t include the tests_regex argument that is also available in the AndroidJUnitRunner.
I’ve recently I’ve been writing a lot of Android UI Tests using TestParameterInjector to support running the same test, but with run with different parameters. I’ve used the support for enum and Boolean values the most, but it also supports String, Int and others. This has been great, but it has also come with challenges in regards to how to run specific tests.
Example Test
Here is an example login test you might find in the src/androidTest directory which would run as an on-device Android instrumentation test. It uses a enum TestParameter of our custom enum TestAccount {SAM, HANDSTANDSAM }. It will run our single tests testLogin() and testSomethingElse() 2 times each (once for each enum value of SAM, HANDSTANDSAM) without us having to write out each one of those method explicitly. 🎉
package com.handstandsam
import com.google.testing.junit.testparameterinjector.TestParameter
import com.google.testing.junit.testparameterinjector.TestParameterInjector
import org.junit.Test
import org.junit.runner.RunWith
@RunWith(TestParameterInjector::class)
class Test(
@TestParameter private val testAccount: TestAccount
) {
enum class TestAccount { SAM, HANDSTANDSAM }
@Test
fun testLogin() {
println("Login with Test Account $testAccount")
}
@Test
fun testSomethingElse() {
println("Something Else with Test Account $testAccount")
}
}This exact code results in the following 4 executions:
- com.handstandsam.Test#testLogin[SAM]
- com.handstandsam.Test#testLogin[HANDSTANDSAM]
- com.handstandsam.Test#testSomethingElse[SAM]
- com.handstandsam.Test#testSomethingElse[HANDSTANDSAM]
Challenge: Computing a fully qualified method name for tests using TestParameterInjector
Normally the fully qualified method name would be com.handstandsam.Test#testLogin which you could send as an instrumentation argument to run that specific test. However in this case of a test leveraging TestParameterInjector‘s custom runner, it computes the unique test names at runtime via reflection.
If we try to explicitly run com.handstandsam.Test#testLogin, it’ll say “no tests found”.
We can use the computed name of com.handstandsam.Test#testLogin[SAM] and the test will be found.
In this simple example it’s hard to see the challenge, but when you mix multiple @TestParameters in a file and change their ordering, the computed fully qualified test names will change and could be long like test[SAM,true,BLUE] or something.
This means that the only reliable way to run the tests in this class without knowing the full name is by the class itself com.handstandsam.Test but that means 4 tests will run (2*2) which is not what we are looking to do.
./gradlew :app:connectedDebugAndroidTest -Pandroid.testInstrumentationRunnerArguments.class=com.handstandsam.Test
TestParameterInjector GitHub issue
Static computation of fully qualified test names is a challenge using TestParameterInjector, but to be fair the library wasn’t made specifically for Android Tests and this issue is more annoying in Android Instrumentation Tests that take longer to execute. TestParameterInjector is widely used for writing tests with Paparazzi to run the same test against multiple device and screen size permutations, but these tests run so fast that it isn’t a real problem to run them all at once.
Leveraging tests_regex 🎉
I’d like to be able to run all permutations of com.handstandsam.Test#testLogin which are:
- com.handstandsam.Test#testLogin[SAM]
- com.handstandsam.Test#testLogin[HANDSTANDSAM]
I could list out every method specified by the class instrumentation argument:
./gradlew :app:connectedDebugAndroidTest -Pandroid.testInstrumentationRunnerArguments.class=com.handstandsam.Test#testLogin[SAM],com.handstandsam.Test#testLogin[HANDSTANDSAM]
SOLUTION – Use tests_regex
Using the tests_regex argument allows us to use syntax like …testLogin\[.+]$ which will match all test permutations for the testLogin method 😀.
Full example command using test_regex
./gradlew :app:connectedDebugAndroidTest -Pandroid.testInstrumentationRunnerArguments.tests_regex="com.handstandsam.Test#testLogin\[.+]$"
The above command is pretty gross to write, but we’ve been able to generate it out via a Custom IDE Plugin (not generally applicable, so not open sourced) it doesn’t matter how verbose it is, as long as it works!
Conclusion
This tests_regex instrumentation argument has been there for years, but I never knew about it, it hasn’t been blogged about and it’s not in the current Android Developers command-line documentation. Regexes are not my favorite thing to write, but are made easier to figure out using regex101.com. I can foresee other uses of the regex command outside of Parameterized Tests, especially when combined with custom IDE tooling.
I hope you now know about this tests_regex argument and let me know how you’re using it on Bluesky!
