Android JUnit results in Hudson

A little while ago I wanted my unit test results for my Android apps that I build with Hudson. I tweeted about it, but forgot to follow up with some more details, so here it is.

Since I the default android testrunner does not output it's format in a xml suitable for Hudson, I decided to dig around and after looking at source for InstrumentationTestRunner I came up with this.

The com.neenbedankt.android.test.InstrumentationTestRunner extends the Android one so that an xml of the test results is written to the device when running the tests. To use this in your Hudson build you only have to take these simple steps:

Add the testrunner to your project, and reference it in you AndroidManifest.xml (see here)
Startup an emulator in your Hudson build using the android emulator plugin and launch adb to run the tests
Pull the generated xml report file from the device and put it somewhere in your build dir so that Hudson can find it.

If all goes well, Hudson will pick up the results and use that to show the JUnit results.

For my own little projects it seems to work fine, although it I've implemented the bare minimum to get something working in Hudson.

Jul 14, 2010

VJ said...

Hi, this is excellent! I am using andriod junit testing and I am trying to integrate into hudson for nightly reports. But I have a question. For automated nightly builds. How does Hudson know to pull the xml file off the device every time it builds? I don't want to have to manually to this for every build.
Also, is it possible to avoid pulling the xml file from the device and just writing the xml file in our home directory. I tried editing:

startJUnitOutput(new FileWriter(new File(getTargetContext().getFilesDir(), JUNIT_XML_FILE)));

with variations of this:

new FileWriter("TEST-all.xml")
new FileWriter(new File("/home/vj/Desktop", JUNIT_XML_FILE))
etc.

But nothing works, i keep on getting java.io.filenotfoundexpection.

Could you help me solve this problem please?

Oh and I am not using the android plugin, as long as my emulator is up, the tests run on the emulator.

Also, I am new to all of this. So baby steps would be much appreciated :)

And also, where is hudson's build directory? You mention at the very end to pull the xml report and put it in the build directory where hudson can find it...
thanks in advance!

Hugo said...

OK, here goes :)
The path in the code is the path on the actual device, or in this case the emulator. That's why setting the path to your home dir doesn't work, and that's also the reason that you need to pull the file off the emulator using "adb pull".

I highly recommend the Android emulator plugin, I see no reason not to use it. It makes starting up emulators much easier, sets some variables which you can use with adb and makes sure to shutdown the emulator too.

I'm running the tests from ant using the standard Android build scripts and I set test.runner property to com.neenbedankt.android.test.InstrumentationTestRunner using the ant options in Hudson. That will make the build use my testrunner and run the tests.

Then I have a build step to get the file from the device, which is in my case "adb -s $ANDROID_AVD_DEVICE pull /data/data/com.neenbedankt.enginewatch/files/TEST-all.xml $WORKSPACE/test-results.xml"

$ANDROID_AVD_DEVICE is set by the Android Emulator plugin and ensures that I'm connecting to the right emulator. The path to the file is basically /data/data/<yourpackagename>/files/TEST-all.xml and $WORKSPACE is set by Hudson and resolves to your Hudson workspace dir.

Then you check "Publish JUnit..." in Hudson and enter "test-results.xml" for the file.

That should get you started right? :)

Jul 15, 2010

O.M.G.
Hugo, I LOVE YOU!!!!!
thank you so so so so much! It worked!! I can barely contain myself!
And thank you for the speedy reply!

And for anybody out there. You write the second part in the "invoke shell" part

Thank you again :)

Jul 16, 2010

Bob Mac said...

Great stuff, thanks. Any reason you couldn't run the tests on a phone, then pull the results off?

On a related note, in Eclipse I find that the JUnit view only populates testrun results if the tests are run on the emulator - for the phone I have to go to the log (stdout). Just wondering if others have seen that. Thanks again. Bob Mac

Sure, this will also work on a real phone, but I guess it has to be rooted to get the file from the data dir. For my purposes I want to run it automated in Hudson anyway, so for me it doesn't make sense to run it on an actual device. Writing to the sdcard would also work, but that would require additional permissions in the AndroidManifest.xml.
Running tests on the device from Eclipse are also a little bit flaky for me, sometimes they just don't start, but I can run them. Eclipse runs them twice actually, once to discover which tests are there and then the real test run.

Jul 19, 2010

Thanks Hugo - that makes sense. One other question - can you please make your build.xml available? The android cmd didn't generate a "run-tests" target as far as I can tell. Thanks again.
Bob Mac

reggieL said...

Hey Hugo,
This is so helpful. Thank you.

Instead of receiving an email for each test project I build, I would like to receive one email with all the test project test results. What's your take on this? Any ideas?

@Bob, I think it depends on the version of the SDK you are targetting. My build file isn't particular exciting, just what the Android sdk generates (described here: http://d.android.com/guide/developing/other-ide.html#CreatingAProject). For different versions of the SDK the ant task includes different macros, which is a bit annoying.

You can build your app against one sdk version and run the tests with another if you like.

@Reggie, It all depends on what you define as a single job in Hudson. As far as I know the emailing stuff is per job, which make sense because you most of the time want to know if the build breaks, not when all is fine. It depends on your usecase I guess. You should ask this on the Hudson list to get a better answer, maybe there's a plugin available to do what you are looking for?

Jul 29, 2010

cscrazy said...

Hey Hugo, great stuff!

When you run your Hudson automated tests, how do you deal with unlocking the android emulator without changing the app code/adding permissions to the app's AndroidManifest.xml?

Thanks Hugo

Actually, never mind. With the new version of the Hudson android emulator we have the option of not showing the emulator window. this takes care of having to unlock the emulator

Apr 30, 2011

Kyle said...

Thank you SO much. That was incredibly helpful!

Jan 31, 2012

Karthik said...

Thanks Hugo..It worked Big time..Thanks big time..!

code.neenbedankt.com