January 16, 2004 - Continuous Integration, Continuous Agitation

With automated tests, like money in the bank, the joy should be the using more than the having. But unlike money, you can use the same test again and again, and we've found that being a spendthrift with our cpu cycles is the best way to get the feedback we need to drive our product quality.

Starting with best practice

Like a lot of people I first learned of the concept of a nightly build when reading Steve McConnell's classic Rapid Development. That was years ago, but it is still true that if you're looking to build a product that is a great place to start. For months and months now we've been doing a nightly build and test cycle using a home-grown harness. This system uses perl as glue to invoke Ant scripts, parse the results, generate html, update a website and send email to the team. There's a web UI to invoke build and test suites on demand in addition to our regularly scheduled build/test cycle.

This nightly regression email has become of a staple of life at Agitar, to the point that even the CEO takes a look at them when he gets in! Of course it is probably more effective that the developers start their day by checking the results, fixing any issues that came up as the first order of the day. The result has been a virtuous cycle where our QA team spends less time dealing with broken builds, freeing them to develop more test suites, which in turn makes our nightly build process that much more effective.

Making best even better

A more recent addition to our toolkit is CruiseControl and the practice of Continuous Integration. As described by Martin Fowler, continuous integration is a practice that leverages both the investment and the philosophy of the nightly build, and asks the question "why only once a day?" The real limiting factor should be the length of your build/test cycle rather than the calendar.

Having already put the nightly build in place, the introduction of a "cc build" was painless. CruiseControl uses our existing Ant script and picks up the results of our existing JUnit tests. It also offers a very flexible design so that we've been able to add support for Agitator and customize the output to our liking, including integration with jCVS.

This new cc build has become the focus of our developer testing effort, in the same way the nightly build was the focus of our QA testing effort. As our developers write code they create tests that run as part of the cc build. Every time they check in, no matter if it is a change to code or tests, CruiseControl pulls the updates, compiles, runs our developer system tests, agitates the code, generates a dashboard for any classes with failing assertions, and then sends email with the build results. As a result we now get feedback within an hour that previously come in the next day.

Addicted to feedback

An hour is pretty good — pretty darn good actually — but it wasn't long after the introduction of the cc build that some developers came forward with an observation and a request. They noticed that sometimes if they checked-in code with a compile error they were notified almost immediately. But other times, if CruiseControl was already busy, they might not get notified for an hour. Was there any way they could get a quick sanity check on code without having to wait an hour?

The solution was to install CruiseControl on a second machine and create a new Ant target that just pulls the changes, compiles all the code, and runs our smoke tests. This cycle typically takes about 2 minutes, meaning that after a check-in the developer can expect email reassurance (or blame) in their inbox in under 5 minutes.

This quick feedback has really taken hold, to the extent that developers won't leave the building after a check-in until they get the email reassuring them that they didn't break the build totally.

Continuous feedback, continuous improvement

We're extremely pleased with the infrastructure we've grown and feel like we have a nicely tiered approach to the tests we run and the corresponding feedback cycle:

Looking back we feel we did things a bit out of order. For our new project we are starting the process with CruiseControl. Since the codebase is small our full compile/agitate cycle is only a few minutes. As we progress we'll fold in the nightly build with QA tests on multiple systems, and then if the compile/agitate cycle gets longer than some threshold — perhaps 15 minutes — we'll define a new quick subset to get that quick feedback.

One practice we'll be keeping the same is the lack of formal consequences for breaking the build. As Alistair Cockburn observed, "people are generally interested in being good citizens". We don't need to hound them or punish them, but rather make sure they get the right feedback in a timely manner so they can act on it.

Posted by Jeffrey Fredrick at January 16, 2004 01:14 PM

Trackback Pings

TrackBack URL for this entry:


Post a comment

Remember Me?