We recently switched from Cobertura to JaCoCo as the test code coverage tool for our services. After some initial troubles with Powermock the tests were running OK and coverage results were coming back as expected. The numbers looked good too. That was the end of that in terms of official tasks that needed to be done for this switch but I was still curious about any performance or coverage differences between the two tools, so I dug a little deeper...

I reverted to the old Cobertura configuration for Maven and ran some builds with test code coverage enabled. I also ran the same tests with the new JaCoCo configuration. The numbers surprised me!

For these tests I disabled Cobertura's report aggregation since that's not being done with JaCoCo. Both tools were generating HTML and XML reports. JaCoCo was set up with offline instrumentation to ensure that Powermock enabled tests were picked up.

These were the times...



That's quite the difference! Cobertura covered builds were taking more than twice the time to complete! This was not something I paid much attention to while we were still using that tool. On slower machines (not everyone uses a 2.9Ghz Core i7 Mac with 16Gb of RAM) this difference in time would be even more profound.



So I was sold on JaCoCo in terms of performance, it was fast! Next I wanted to compare their coverage reports. Right off the bat it was obvious that each tool reported very different statistics and so it was hard to compare them...

JaCoCo -

Cobertura -

In some places JaCoCo showed more coverage, other places Cobertura showed more. It was a little confusing. Looking over the numbers it seemed that each tool was calculating the number of lines per class differently, which explained some variation. Other parts were harder to explain like huge differences in branch coverage and also differences in line coverage.

Then I noticed that Cobertura reported line coverage whereas JaCoCo reported instruction coverage! Looking into some of the classes that were missing coverage I could see this...

JaCoCo -

Cobertura -

It looked like JaCoCo didn't consider a line as covered unless all the branches on that line were fully covered. Cobertura considered a line covered even if not all of the branches were fully tested.

So which tool do I think is better? In terms of productivity I think I'd have to go with JaCoCo. In terms of setup, Cobertura wins. In terms of the generated reports - I like a bit of each one so they're on par here though I do like Cobertura's package layout (JavaDoc style). In terms of coverage - it looked like JaCoCo was showing more accurate results. In the end productivity and accuracy won since setup is a once-off activity and the reports from both tools show the same basic information. So JaCoCo is my choice then.

-i