Parallel concurrent execution of instrumentation tests

[TheKeeperOfPie]

I realize I'm using experimental APIs on top of experiment APIs, but does anyone know if this works with JUnit 5's parallel execution in concurrent mode?

The actual running of the tests seems to work just fine. If I log the end of my test methods, I can verify they all actually execute on device when running through gradlew and Studio, but the test reporter spits out Test run failed to complete. Expected 4 tests, received 1.

So maybe support is already there, but the test reporter doesn't know how to handle concurrent test results. I might also be holding it wrong.

I tried to add support here in case a repro is needed: TheKeeperOfPie/ArtistAlleyDatabase@ecb900b

You'll have to toggle the enabled property here. And execute EntryDetailsAddTest. The repo has a few setup steps, and you might also need to delete gradle/verification-metadata.xml because I've had trouble bringing it up on a fresh project with dependency verification working.

[TheKeeperOfPie]

Okay, so it seems AndroidJUnit4 doesn't support reporting test results out of order. Parallel works if you edit the AndroidJUnitPlatformRunnerListener to collect RunNotifier events and only send them once a test finishes, ensuring that each test method gets all of its events reported in full before another method's events.

The downside to this is that method timing reporting is now lost, since everything "finishes" instantly. But it looks like the tests themselves do execute in parallel.

Unfortunately this requires editing a lot of hidden library code, so there's no easy patch for consumers. I might try and fork to add support, or maybe there's an easy way to detect if parallel tests are running and offer an option to do this collection logic.

This could also just be a bug in the AndroidJUnit4 reporter, but since there's no supported way to run parallel tests in the 3P world, it probably won't get fixed.

[TheKeeperOfPie]

Actually, it might be simpler than I thought. This seems to work. I had to use the snapshot version of the plugin to avoid a builder configuration check.

android {
    defaultConfig {    
        testInstrumentationRunnerArguments["runnerBuilder"] =
            "com.example.AndroidJUnitBuilder"
    }
}

class AndroidJUnitBuilder : RunnerBuilder() {
    private val builder = AndroidJUnit5Builder()
    
    override fun runnerForClass(testClass: Class<*>): Runner? {
        val runner = builder.runnerForClass(testClass) ?: return null
        return object : Runner() {
            override fun getDescription() = runner.description
            override fun run(notifier: RunNotifier) = runner.run(ParallelRunNotifier(notifier))
        }
    }
}

class ParallelRunNotifier(private val notifier: RunNotifier) : RunNotifier() {

    class Failures(var assumptionFailure: Failure?, var testFailure: Failure?)

    private val events = mutableMapOf<Description, Failures>()
    private val lock = Any()

    override fun fireTestStarted(description: Description) {
        events[description] = Failures(null, null)
    }

    override fun fireTestFailure(failure: Failure) {
        events[failure.description]!!.testFailure = failure
    }

    override fun fireTestAssumptionFailed(failure: Failure) {
        events[failure.description]!!.assumptionFailure = failure
    }

    override fun fireTestFinished(description: Description) = synchronized(lock) {
        val event = events[description]!!
        notifier.fireTestStarted(description)
        if (event.assumptionFailure != null) {
            notifier.fireTestAssumptionFailed(event.assumptionFailure)
        } else if (event.testFailure != null) {
            notifier.fireTestFailure(event.testFailure)
        }
        notifier.fireTestFinished(description)
    }
}

[mannodermaus]

Hey, thanks for the research and apologies for the late reply. I haven't explored parallel test execution with instrumentation tests yet, but your latest snippet seems like a promising starting point. Thank you for that! I'll try it out and will let you know.

[mannodermaus]

@TheKeeperOfPie Thanks again for being patient. I was able to put in some work here and managed to make the reporting of parallel instrumentation tests better. Please feel free to check out the results in the latest 1.4.0-SNAPSHOT and let me know if you encounter any problems.

[TheKeeperOfPie]

I got around to testing this and it seems to run the tests properly (tests expected matches tests actual), but unfortunately the runner still reports that the suite failed. But if I open the test report, it shows 100% success rate, so I don't know where the disagreement comes from. It's reproducible with ./gradlew test connectedAndroidTest in my same app repo, but I'll try and get a narrow scoped reproduction when I get the chance.

Regardless, thanks for taking a look at this. At the very least this lets me remove my workaround and I can rely on manually inspecting the test report to determine if all tests passed. And Studio integration works great now.

[mannodermaus]

Thanks for the feedback! Glad to hear that it (mostly) works for you. Let me try to look into the suite failure report, it's not something I have encountered while working on this. Hopefully your sample repo can give some insight. 🙏