Homepage ->  Individual Weblogs -> Kevin Lawrence  -> Failing tests shouldn't always break the build

August 11, 2006 - Failing tests shouldn't always break the build

There are many reasons why you might not want to fix a failing test right away. Maybe it's an acceptance test for a feature that you haven't written yet. Maybe it's a regression that it's just not practical to fix right now.

But what to do with that failing test?

If you leave the test failing it will obscure other failures. If you leave the build breaking for long enough, people will assume that build is meant to be broken and break it some more. You need to get rid of that failing test somehow.


Most developer's first instinct is to comment out the test - I know it was mine - but a commented out test is pretty much gone forever. You might as well just delete it. Don't believe me? Go look at the last test you commented out. Had you forgotten about that one? Thought so.

NUnit (and now JUnit4) has an @ignore annotation. That's a little better than deleting the test but you have to remember to un-ignore it at some point and it's as easy to ignore a hundred tests as it is to ignore one. Unless you have the discipline to go back through all the ignored tests periodically, they will probably stay ignored forever.

Someone on the TDD list suggested putting an expiry on the @ignore so that, at some point in the future, it will turn back into a failure. What then?

One trick I have been using recently is to associate a bug number with every failing test that can't be fixed immediately. It's especially useful for acceptance tests.

For example:


public void testHomePageShouldShowTheVersionNumber(){
bug(8674);

onPage("index.html")
.theTextAt("//span[id='version']")
.shouldMatch("4\\.0\\.0\\.[0-9]*");
}

The test will still fail if the version number code is broken, but the call to bug() writes a little log file that cross-references bug numbers with test names. If 8674 is open, the test will still fail but I have a BugVerificationTask that runs under ant and queries Bugzilla. It will only cause the build to fail if there is a failing test without an open bug.

[no doubt I could have used an attribute/annotation to the same end but I need to support JDK1.4 too]

Once the feature is implemented the test will pass but, if anyone ever introduces a bug that causes the test to fail, we have a range of options :

You probably already annotate your tests with bug numbers either in the checkin comments or in javadoc for the test. Having it right there in the code makes it easy to see the whole history of a test. Why was it created? Why did it fail? Why did it break last time?

As an added bonus, you can build some pretty nifty reports on top of the Ant JUnit output. I have one that groups the test results into passing tests, known failures and failures-with-no-open-bug. It lists all the bugs for each test with their failures. Each bug number is a link to bugzilla and its tooltip shows the bug summary.

The key innovation was to separate the failingness from the importantness. Some failures are important right now. Others less so.

If a failure is not gonna get fixed immediately we don't want it to turn the lava lamp red because, as we know, red lights attract bugs. Or something.

Now if only I can get the people who make the test runners to put something like this into Eclipse and IntelliJ. David? Kent?


Posted by Kevin Lawrence at August 11, 2006 03:07 PM


Trackback Pings

TrackBack URL for this entry:


Comments

I'm listening. I think it's really cool. I've spent five minutes thinking about the Eclipse logistics. By my point of view, by the time the Eclipse bar goes red, it's too late--the bugs are gathering. So then, do we call out to bugzilla in realtime? Cache? There's a good answer here, but I'm not smart enough today. Maybe tomorrow.

Posted by: David Saff on August 14, 2006 03:34 PM

There should be more than just two states for test results:
Pass, Fail, Expected failure (the case you mentioned here), Fixed failure (expected failure which passed, implies that bug is fixed), Skipped test (e.g. does not apply for this platform), and Testing Failure (fixture blew up).

You would need to mark if a test was expected to fail, or if a test was to be skipped.

Unfortunately, I have not come across a tool which uses these test result states.

CP

Posted by: Chris Pousset on August 15, 2006 11:33 AM

Another problem using that approach is that you are dependant on being able to get an exact state of your bug reporting system at any point in time, as this may be necessary if you have to run (and build) your product using a CVS snapshot that may be 1 year old.
Basicly you need to have all entries/changes in your bug report system being checked into the same repository as your normal code unless you want to accept failures during execution of unit test.

Posted by: Henrik Gammelgaard on August 16, 2006 04:35 PM

It is definitely a trickier problem to solve in a GUI test-runner. Perhaps it could be implemented as some kind of filter? Exclude known problems? Include current release only?

Posted by: Kevin Lawrence [TypeKey Profile Page] on August 17, 2006 02:14 PM

Post a comment




Remember Me?