Who should read this article: If you develop for iOS and you unit test, this is for you. I compare GHUnit with the revamped, Xcode 4-integrated OCUnit/SenTestingKit, and provide recommendations based on various criteria.
In February, I wrote about unit testing on iOS using GHUnit (under Xcode 3.2.x).
After the release of Xcode 4, our reader Christian (thanks!) responded with an important question: “Xcode 4 improved unit testing support with OCUnit, so is GHUnit still a relevant recommendation?”
Well, Christian, that is one tall order of a question.
However, I’m in just the position to research an answer:
- I’m 33,000 feet over the Pacific Ocean, next to Kamchatka (pondering attacking Alaska on my next turn after I turn in my set).
- From Tokyo, it’s 13 hours to New York.
Let’s do this thing.
** 11 hours later (no, really) **
The Up-Front Conclusion
OCUnit/SenTest is now good enough to deserve your attention. I recommend it for all new Xcode 4 projects. I still recommend GHUnit for any project where GHUnit is already in use.
OCUnit still has a few drawbacks (log output/readability, inability to run & re-run individual tests), but those are now outweighed by the time saved by its now-proper integration into XCode. However, it’s not SO compelling that I recommend switching immediately.
GHUnit shouldn’t die with the XCode 3.2.x era. It is a different offering with some great features. Particularly, if you like easy-to-read output, or have a lot of resources to load in your tests, GHUnit may still be your choice. It also has some handier assertion macros.
If you decide that OCUnit’s limitations are a problem, you can switch to GHUnit at that point; GHUnit will run the tests in your SenTestCase subclasses – which is why I also recommend that new test cases for existing projects using GHUnit should be written as subclasses of SenTestCase, not GHTestCase.
If and when you migrate later, you’ll have less to re-write.
Our Criteria
The following list is in order of importance (most to least) of what matters most (to me) when unit testing. I used this list as criteria for my judgement.
- How easy is it to create a new test class?
Lazy programmers don’t write tests if the process is painful. - How easy is it to write a useful unit test method?
One that’s actually going to catch a bug, not just some ennui for the sake of example. - How easy is it to debug a test method?
When it does fail, I need to be able to quickly figure out why so I can fix it (the whole point?). - How easy is it to interpret the log/results of a test?
If a method fails, I probably have some debug output that should help. - How easy is it to run the test suite?
Tests should be run often to pick up on failures quickly. Running tests frequently also keeps the programmer’s head in the tests. - How easy is it to automate running the test suite?
Automated testing is even better; I can go get a coffee or a beer, depending on the time of day. - How easy is it to set up unit testing for my project?
- How well do my tests integrate with my toolchain?
If you care about different things, you may not agree with my conclusions above; I provide a discussion of each point below, however, so you’re welcome to draw your own conclusions based on your needs.
History
If you know why OCUnit/SenTest sucked, skip to the Comparisons section.
That said, I think this section is fun (at least it was fun to write, maybe I’m loopy due to being in a plane for this long).
Under XCode 3, it was not possible to set a breakpoint in an OCUnit test case*. Unit tests ran as a build script, so when a test failed, XCode reported it in the build results as a build error in a hard-to-read, very uncomfortable place (like the back of a Volkswagen).
A testing purist could argue that the test served its purpose: “for a given input, the test compared output to the expectation, and failed”.
Not enough for me. I am demanding; my unit tests must help me track down the root cause, and fast. Otherwise, you’re stuck littering your code with NSLog statements trying to figure out where things are going awry. The purists here might say “your unit tests are testing too much then”, to which I have a separate -rant- post that I want to share with them. Leave a comment if you actually side with the purists here.
(*Actually, you could set up XCode to run the debugger on unit tests; setting it up required not a trivial number of build/environment settings. I stumbled across this how-to written by Scott Densmore when researching this post.)
The rest is history: lazy developers everywhere denounced SenTest and sought alternatives. Google Toolkit for Mac and GHUnit, as I understand it, came into existence precisely because of these problems with SenTest.
Comparisons
- How easy is it to create a new test class?
OCUnit: 10/10, piece of cake. Press Command-N to add a new “Objective-C test case class” template to your project.
GHUnit: 8/10, pretty easy. Same process, but there’s no default XCode template, so it requires a copy/paste from another file, or requires you to install another template in Xcode4 (also easy).
OCUnit is better for the built-in template, but if you set up an Xcode template for GHUnit, it’s a wash.
Result: Tie.
- How easy is it to write a useful unit test method?
OCUnit: 8/10, pretty easy. OCUnit’s test bundle is injected into your application code before execution, which has some side effects to which you have to pay attention.
GHUnit: 10/10, great. There’s no artificial distinction between logic & application tests. Also, it has -setUpClass and -tearDownClass methods.
“How easy is it to write a test” probably can be broken down into a few bits – assertion macros, class structure, “access” to the real program code.
Foremost, both frameworks have essentially the same assertion macros. GHUnit has a few more helper-like macros (particularly for C structs), but you could easily copy-and-paste them or write your own.
As for class structure, in certain tests, you may need to setup a few mocks and external resources. For example, recently I loaded a large PLIST into an immutable NSDictionary using GHUnit’s -setUpClass.
If I was using OCUnit, I’d have to put resource-loading code into -setUp, which is called for each test, slowing everything down. Slower tests = programmers run less frequently = bad for business.
Finally, what you can access. In GHUnit, anything. OCUnit wins the “Most Improved” award in this category. Previously, it was not possible to run “application tests” on anything other than an iOS device. Apple seems to have eased this restriction; the iOS simulator now runs application tests as well as logic tests!
(In case you don’t know the difference or why it matters, Google around a bit.)
Result: GHUnit, by a small margin
- How easy is it to debug a test method?
OCUnit: 10/10 – great. Breakpoints work, finally!
GHUnit: 10/10 – great. Breakpoints work, still!
Again, OCUnit jumped leaps and bounds. This capability along with the simulator-based application unit testing have made it a contender.
Result: Tie
- How easy is it to interpret the log/results of a test?
OCUnit: 4/10 – a litte painful. Log output is console only, but you get some help from the XCode 4 interface.
GHUnit: 8/10 – good. Test results are in a UITableView, color-coded, filterable, and are navigable.
Evaluation:
I’ve never liked the way Xcode has handled unit test failures; Xcode treats unit test failures as a build failure. While I appreciate that it’s part of the build process, in Xcode 3 it was downright painful. In Xcode 4, Apple has improved this greatly by putting the failures on the left navigator; clicking a failure directs you to the STAssert responsible for the failure. Useful.
One “feature” that annoys me is that Xcode 4 will highlight the line of code with the failing assert in red. Good intentions, but the red bar stays there until you re-test and the tests pass. Again, the purists are saying “GREAT”, but if it’s a STFail() stub simply because I haven’t written the test yet, and am busy over on the actual code, I don’t want to pressCommand-B and see a red icon at the top. I’ll get confused and think my build failed. Who knows, maybe I’ll get used to this – it’s personal preference I suppose.
In this way, I like the way GHUnit keeps unit testing separate from the rest of the build process. That, and the log output is MUCH easier to read and understand.
Result: GHUnit, by a reasonable margin.
- How easy is it to run the test suite?
OCUnit: 8/10, great – built-in to Xcode 4′s workflow.
GHUnit: 7/10, good – but have to manage second target.
One new feature of XCode 4 is the “scheme”, integrating debugging, testing, profiling, and release into a coherent entity. There’s a new build setting that allows you to run your tests immediately following a build (alternatively, press Command-U to run your unit tests manually).
Using GHUnit, you need to switch your target to the GHUnit target, build, and run. This also means you’re building twice, unlike OCUnit, which re-uses the build, saving time.
I’d be doing GHUnit a disservice to not mention two awesome features – you can run each test individually. If just one thing is failing, why run everything? You can also re-run tests, which is useful if you’re in a situation where a test could pass or fail depending on non-compiled resources (e.g. we used GHUnit to run integrity tests on a text-based file format; if it fails, you fix the file, and re-run the test while GHUnit is still running).
AFAIK, OCUnit doesn’t offer these.
Result: Tie, each can be better for different types of testing.
- How easy is it to automate running the test suite?
OCUnit: 7/10 good; can be great with a few hacks.
GHUnit: 2/10 almost impossible
GHUnit: x/10 unknown (I’ll explain this in a second).
I mean a continous integration server (CI) when I say “automate”.
CI allows your team to publicly shame any memmber who commits a broken build and/or failing tests. Shame goes a long way in reducing Lazy Programmer Syndrome (a.k.a. “I’ll do it later [never]“).
Many people test their builds with CI by having the command line tool xcodebuildbuild their projects whenever anyone commits anything. For unit testing automation, it usually happens right after the build – so we need to use xcodebuild on the command line again.
EDIT
Previously, I’d given GHUnit a low score for CI. Reader Rusty reminded me in the comments that it is indeed possible – I just didn’t know how. For now, I’m leaving it as x/10 because I haven’t done it personally yet, so it’s not really fair to judge. However, fellow iOS dev Robert has, and commenter Mohamed brought that to my attention. See this article about setting up GHUnit with Xcode 4. Robert even has a sexy video.
/EDIT
On the other hand, OCUnit is also a separate target, but it’s a special type of target in Xcode’s eyes; it’s a “test bundle”. Accordingly, after building a test bundle, Xcode 4 is already set up to run the tests with shell scripts.
Note that I didn’t give OCUnit 10 out of 10, though.
When researching this topic, I first found this post by Jonah explaining how to run xcodebuild on the command line to run logic tests only (not application tests).
Jonah’s method requires setting up a new “scheme” for your unit tests, which sounded a lot like how GHUnit is set up. To me, this is a step in the wrong direction for OCUnit, as it’s gaining a lot of points with me via tight integration.
However, the end of his post gave some great hints on how to move forward, though, and I believe we’ve solved both issues: you don’t need a separate scheme, and you can run application tests as well as logic tests from the command line (it has been a LONG plane ride).
Because not everyone wants to do command line builds of their Xcode projects, I’ll put the instructions up as a separate post tomorrow.
Result: OCUnit, but not by much, because the functionality doesn’t come out of the box.
Result: Needs to be re-evaluated based on the new information above.
- How easy is it to set up unit testing for my project?
Of course OCUnit wins this, it’s built-in to Xcode. When you set up a new project, it’s a tick box. If you’re adding to an existing project, it may not be as trivial; in that case, I’d say either GHUnit or OCUnit is about equally difficult (or easy). I have heard some anecdotal buzz on StackOverflow that adding OCUnit/SenTest into an existing Xcode project still isn’t trivial, even in Xcode 4.
GHUnit takes me less than 10 minutes to install with these instructions.
Result: Tie for existing projects, OCUnit for new projects.
- How well do my tests integrate with my toolchain?
I can’t really effectively rate these numerically, as everyone’s toolchain is different.
OCUnit is open-source, as is GHUnit. That said, precisely because of OCUnit’s integration into Xcode, there’s some magic going on behind the scenes. You’re using a precompiled SenTestingKit framework to run those tests.
When you use GHUnit, on the other hand, you have to build the framework after checking out the repository. If you need to change something to match another tool in your kit, GHUnit is going to be easier. For example, GHUnit comes with JUnit output capability.
Result: GHUnit
Extra Thoughts
There are a few points I haven’t covered above.
- One of GHUnit’s advantages is that it is built on top of SenTest and GTM, so it can actually run your OCUnit SenTestCase test classes out of the box. If you’re writing test code that could end up being compiled in someone else’s project, that’s worth knowing. Unfortunately for GHUnit’s maintainers, it makes me more likely to choose OCUnit, because I know GHUnit will run it as well.
- Apple has neglected and broken unit testing before (specifically, in iOS 4.1 SDK). They may do it again. Even though I’m leaning towards OCUnit for the near future, I will keep my eyes on GHUnit.
- If you work in a team and not everyone is running Xcode 4, stick with GHUnit, by all means. Xcode 4 takes a bit of processing power (it grinds my MacBook 13″, often, even with max’ed RAM), at least far more than Xcode 3. Not everyone should run out and upgrade (particularly if you have an older MacBook or low-powered MacBook Air).