I just wrote a page on our internal wiki with our policy for dealing with build failures. We thought others might find it interesting so I am sharing it here (the links will be broken for obvious reasons).
If the build fails, fix it.
directly from wiki ....firstname.lastname@example.org that has "Build Failed" in the subject. If it was not your fault, look at the modifications list in the cc email to find the culprit. Guide them gently towards Step 1. If it is not obvious who the culprit is - or if the culprit is unavailable - fix it anyway. email@example.com and explain briefly that you are/are not fixing it. If it was not your fault and the light is still on after several minutes, inquire gently whether anyone is dealing with the broken build and when it will be fixed.
bug()in the failing test. The build will not fail for a failing test that has an open bug. The build generates a report that separates known issues from new failures. Bugs that are scheduled for the current release will be shown in red. Unit tests should be fixed immediately. http://cc-unit-test:8080/cruisecontrol/ and a failure in any of them will light the light. http://cc-builder:8080/cruisecontrol/ http://cc-builder:8080/cruisecontrol/ A failure in any of them will light the light - unless there is an open bug (see above).
Posted by Kevin Lawrence at August 24, 2006 04:20 PM
TrackBack URL for this entry:
Interesting policy - something I think we should adopt more rigidly too.
We have recently had the unfortunate problem of failing acceptance tests as we have rapidly built a suite of tests for our main product and some are a bit unstable. This can sometimes mask the genuine unit test failures in that product or other products built by cruisecontrol. Or, worse, a behavioral change in the software that the acceptance tests have highlighted, but that are ignored because people just think "the acceptance tests are always failing...".
How does your bug() call work? I am interested in implementing something similar. At the moment we have a suite called "failingTests" that bad acceptance tests are added to and are not run as part of the CC build. I'd much prefer the simpler mechanism of a method call in a failing test to bug() that was still logged and tracked in the CC build reports...
Posted by: Patrick Myles on September 2, 2006 05:44 AM
I have bug() write the bug number and test name to a file bugs.txt.
Then I have a custom ant task that generates an xml report by merging bugs.txt with the TEST-xxxx.xml files that the ant JUnit runner generates. It annotates the bugs with data from bugzilla. I have an XSLT file that generates an HTML report with separate sections for passing tests, failures wth open bugs and new failures.
Finally I have another ant task that reads the XML file and causes the build to fail iff there is a failure with no open bug.
Does the annotated XML report that you create and your XSLT file work as part of the cruisecontrol transforms, or is that HTML report independent of cruisecontrol's output?
So I guess you don't use the ANT junit runner to do the build failing, but rely on your last ant task to do that instead. I hadn't thought of doing that...
I don't suppose you want to share some of your code do you? ;-)
Posted by: Patrick Myles on September 5, 2006 09:17 AM
The xml report generator and verifier are separate ant tasks. The XSLT runs in ant and generates a report that I copy into CC's archive directory.
The verifier also writes a summary to ant's logger with a link to the report.
I have an index card on my desk that says
integrate the xslt transform into CC's email
and another that says
write an ant task that does all the above in a single task.
I'd be delighted to share the code when I get a chance to disentangle it from the rest of my build code. Any day now ;-)