Igor Kromin |   Consultant. Coder. Blogger. Tinkerer. Gamer.

We recently switched from Cobertura to JaCoCo as the test code coverage tool for our services. After some initial troubles with Powermock the tests were running OK and coverage results were coming back as expected. The numbers looked good too. That was the end of that in terms of official tasks that needed to be done for this switch but I was still curious about any performance or coverage differences between the two tools, so I dug a little deeper...

I reverted to the old Cobertura configuration for Maven and ran some builds with test code coverage enabled. I also ran the same tests with the new JaCoCo configuration. The numbers surprised me!

For these tests I disabled Cobertura's report aggregation since that's not being done with JaCoCo. Both tools were generating HTML and XML reports. JaCoCo was set up with offline instrumentation to ensure that Powermock enabled tests were picked up.

These were the times...
 JaCoCo Timing
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:03 min
[INFO] Finished at: 2018-03-06T17:15:37+11:00
[INFO] Final Memory: 74M/768M
[INFO] ------------------------------------------------------------------------
real 1m4.946s
user 4m32.879s
sys 0m27.311s

 Cobertura Timing
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:48 min
[INFO] Finished at: 2018-03-06T17:39:25+11:00
[INFO] Final Memory: 75M/792M
[INFO] ------------------------------------------------------------------------
real 2m49.742s
user 10m33.524s
sys 1m3.292s

That's quite the difference! Cobertura covered builds were taking more than twice the time to complete! This was not something I paid much attention to while we were still using that tool. On slower machines (not everyone uses a 2.9Ghz Core i7 Mac with 16Gb of RAM) this difference in time would be even more profound.

So I was sold on JaCoCo in terms of performance, it was fast! Next I wanted to compare their coverage reports. Right off the bat it was obvious that each tool reported very different statistics and so it was hard to compare them...

JaCoCo -

Cobertura -

In some places JaCoCo showed more coverage, other places Cobertura showed more. It was a little confusing. Looking over the numbers it seemed that each tool was calculating the number of lines per class differently, which explained some variation. Other parts were harder to explain like huge differences in branch coverage and also differences in line coverage.

Then I noticed that Cobertura reported line coverage whereas JaCoCo reported instruction coverage! Looking into some of the classes that were missing coverage I could see this...

JaCoCo -

Cobertura -

It looked like JaCoCo didn't consider a line as covered unless all the branches on that line were fully covered. Cobertura considered a line covered even if not all of the branches were fully tested.

So which tool do I think is better? In terms of productivity I think I'd have to go with JaCoCo. In terms of setup, Cobertura wins. In terms of the generated reports - I like a bit of each one so they're on par here though I do like Cobertura's package layout (JavaDoc style). In terms of coverage - it looked like JaCoCo was showing more accurate results. In the end productivity and accuracy won since setup is a once-off activity and the reports from both tools show the same basic information. So JaCoCo is my choice then.


A quick disclaimer...

Although I put in a great effort into researching all the topics I cover, mistakes can happen. Use of any information from my blog posts should be at own risk and I do not hold any liability towards any information misuse or damages caused by following any of my posts.

All content and opinions expressed on this Blog are my own and do not represent the opinions of my employer (Oracle). Use of any information contained in this blog post/article is subject to this disclaimer.
Hi! You can search my blog here ⤵
NOTE: (2022) This Blog is no longer maintained and I will not be answering any emails or comments.

I am now focusing on Atari Gamer.