It would be handy to have a report of all the unexpected exceptions that get thrown.

I've lost count of the number of times I've heard that. I've even said it myself a few times.

It seemed like it would be pretty easy to do using the dashboard.xml feature of AgitarOne, so I decided to give it a try.

First we generate over a million free JUnit tests and then we get nominated for a Jolt Award.

What a great end to the first year of JUnit Factory!

It is a curious fact that, if you say

Let them eat cake!

at an international gathering, the French-speaking people will have no idea what you are talking about. Even if you say it in French.

A History of Build Systems

In my younger days, before I knew any better, many projects I worked on compiled and published their software manually. You'd type cc and then copy these bits over there and then zip that directory and post it to there. Eventually, we figured out we could write little scripts to automate all the tedious bits and make it less fragile and more repeatable.

One day, I discovered the discipline of daily builds and tools like make and my life got a whole lot better. Make gave us, in Elizabeth's handy phrase "a place to put things".

Elizabeth Hendrickson is a tremendous facilitator and a canny manipulator.

In Influence: The Psychology of Persuasion, Robert Cialdini describes various techniques for making people do things that, if they were thinking clearly, they would otherwise not do because of lethargy, laziness, or because it would offend their better judgment.

One of those techniques is The Commitment Principle which was used on American POWs to great effect by the Chinese during the Korean War.

"Wow!

I have just generated unit tests for some code that would have taken months or years to do manually and I did it in under 30 minutes including registering on your server and waiting for the reply email.

That is truly awesome!"

Getting this kind of feedback is the fun part about having free (as in beer) software up on the web where anyone can try it out. In this case Nick had a great experience with JUnitFactory and let us know with a post to our forum.

Tonight I'll be presenting a modified version of my upcoming Agile 2007 talk, To Catch A Bug, You Have to Think Like a Bug at the BayXP meeting hosted at Google in Mountain View.

I might also show off the new crap4j tool that we have been working on lately.

I'm very excited that with two weeks remaining before the conference we've reached the 100 mark for registrations for CITCON Sydney. This is our fourth CITCON event but our first in Australia, so it is great to see so many people registered. We'll take up to 150 so if you're interested in attending, it's not too late to register. The conference will be July 27th and 28th, Friday night and then all day Saturday.

Btw, if you're not familiar with the Open Space conference format you might want to read this description. If you're still not sure if this format or event is for you, you might want to see what people have posted about past CITCONs on the web, or read the feedback from Dallas earlier this year.

Tomorrow night, I will be giving a talk at the Association of C and C++ Users in Silicon Valley entitled, "To Catch a Bug, You have to Think Like a Bug". This is a new and improved version of a talk I gave at SD West, so if you didn't get to go there, you get another opportunity to check it out. I hope to see some Agitator's there. You can find out more of the details at the ACCU's website.

I am going to be attending sessions at JavaOne this week, and would be happy to meet with any Agitators or testing enthusiasts at the conference, according to Sun's Event Connect tool, I can paste this code and you can link to me in their event tool to set up a meeting.

I also set up a topic proposal for the JavaCamp, unconference that is happening Tuesday and Wednesday nights, on Adding JUnit to the Java Platform.

Here's the blurb for anyone interested:

Many language platforms, like Ruby and Microsoft .NET ship with a unit testing framework as part of the platform. Why not include JUnit in the Java Platform, or at least include it in the JDK? Code quality is a constant sore spot for commercial applications, so it seems like making the tools that contribute to higher quality more widely available will encourage better code. While we're at it, lets put in a code coverage tool as well, so we can see how well we're testing. We already have some profiling and management tools built in, so this seems like a missing piece of the puzzle.

I'd like to discuss this idea, and concerns around improving code quality with developer testing in general.

JUnit Factory has a new feature - project dashboards - and I thought I'd try it out on my bowling code.

Referring to "The Way of Testivus" entry:

http://www.artima.com/weblogs/viewpost.jsp?thread=203994

Morgan Conrad asked: "What is Testivus' wisdom concerning the proper percentage of test coverage?"

Here you go Morgan:

Testivus On Test Coverage

Early one morning, a programmer asked the great master:

“I am ready to write some unit tests. What code coverage should I aim for?”

The great master replied:

“Don’t worry about coverage, just write some good tests.”

The programmer smiled, bowed, and left.

...

Later that day, a second programmer asked the same question.

The great master pointed at a pot of boiling water and said:

“How many grains of rice should put in that pot?”

The programmer, looking puzzled, replied:

“How can I possibly tell you? It depends on how many people you need to feed, how hungry they are, what other food you are serving, how much rice you have available, and so on.”

“Exactly,” said the great master.

The second programmer smiled, bowed, and left.

...

Toward the end of the day, a third programmer came and asked the same question about code coverage.

“Eighty percent and no less!” Replied the master in a stern voice, pounding his fist on the table.

The third programmer smiled, bowed, and left.

...

After this last reply, a young apprentice approached the great master:

“Great master, today I overheard you answer the same question about code coverage with three different answers. Why?”

The great master stood up from his chair:

“Come get some fresh tea with me and let’s talk about it.”

After they filled their cups with smoking hot green tea, the great master began to answer:

“The first programmer is new and just getting started with testing. Right now he has a lot of code and no tests. He has a long way to go; focusing on code coverage at this time would be depressing and quite useless. He’s better off just getting used to writing and running some tests. He can worry about coverage later.”

“The second programmer, on the other hand, is quite experience both at programming and testing. When I replied by asking her how many grains of rice I should put in a pot, I helped her realize that the amount of testing necessary depends on a number of factors, and she knows those factors better than I do – it’s her code after all. There is no single, simple, answer, and she’s smart enough to handle the truth and work with that.”

“I see,” said the young apprentice, “but if there is no single simple answer, then why did you answer the third programmer ‘Eighty percent and no less’?”

The great master laughed so hard and loud that his belly, evidence that he drank more than just green tea, flopped up and down.

“The third programmer wants only simple answers – even when there are no simple answers … and then does not follow them anyway.”

The young apprentice and the grizzled great master finished drinking their tea in contemplative silence.

Translator’s Introduction

In May 2006, an ill-prepared international expedition to the Himalayas lost its way. After two weeks of wondering around, hungry, thirsty, and smelling like inexperienced expeditioners who got lost for two weeks, they stumbled upon the entrance to an ancient cave.

Once inside, they saw a maze of ancient, and messy, cubicles. Each cubicle had a wooden desk, an ergonomically correct bamboo chair, a Dilbert™ calendar, and a strange computer-like mechanical device. In one corner of the office they found barrels of dark liquid which they later identified as early examples of carbonated and highly caffeinated drink and a ping-pong table. They realized that the cave was an ancient software start-up. The oldest one on record. Older even than Netscape.

Among the many things they discovered inside the cave was a note left by one of the programmers. The expedition’s guide, while not very good at guiding, knew how to read the ancient language and translated the note for them:

We have finished the release ahead of schedule – again. All the tests pass, so we are taking the rest of the week off. We are going sailing. Since it’s a team building exercise, we hope we can get reimbursed for it.

The explorers looked at each other in astonishment. Not only had they discovered the oldest software start-up in history, they had also discovered a team of programmers who, apparently, completed their code ahead of schedule ... on a regular basis!

What was the secret of these ancient programmers?

And what had happened to them?

The expeditioners searched each cubicle for clues and found two mysterious booklets. One of them was called "Learn To Sail In 30 Minutes”, which explained the fate of the programmers. You are holding in your hands a translation of the other booklet: “The Way of Testivus”.

Who wrote this mysterious booklet? What is Testivus? Only Google™ knows for sure.

Is the content of this text responsible for these ancient programmers being able to complete projects ahead of schedule?

We can’t be sure, but we believe that the amazing prowess of these programmers was probably due to a combination of the Testivus philosophy, and the consumption of large amounts of the dark caffeinated liquid found in the cave.

Read the booklet and draw your own conclusions.

Alberto Savoia, April 2007, Mountain View, Ca

I came across a nice quote from Ron Jeffries in answer to the eternal question about the color of the tests that result from TDD.

I hosted a JUnit Factory presentation a few days ago (you can watch it online if you missed it first time around) and spent a fair amount of time talking about the Triangle sample in the JUnit Factory demo.

There are a handful of web technologies that I use a lot. It's handy to have a cheat sheet around for when I can't remember whether it's switch-case or choose-when or if-test-else.

Here are a few that I use all the time:

• xml, xpath & xslt
• JSP, EL, JSTL
• Regex

Any more I should know about?

Sometimes an aphorism like “good development depends on good communication” doesn’t really sink in until it hits you upside the head. I experienced the reality behind this particular maxim recently when I expanded the number of developers on an open source project from one developer, myself, to two.

I can code passably well and I am comfortable with public speaking - but there is something about combining the two that makes my brain just completely shut down.

For completeness, I run the characterization tests one last time. As you might expect, there are failures because the behavior of getSecondBall() and needsMoreBalls changed.

I am impatient to be done now, so I'll try to get through the code for spares quite quickly so that I can review my findings.

Someone asked me how the characterization tests fared after such an extensive change. After all, I added new methods, new behavior to existing methods and I refactored extensively.

With the code for strikes written, it's time to run the acceptance tests to see if they agree that we are done.

Tomorrow morning, I'll be giving a talk at SD West 2007 on developer testing. It is a a very opinionated look at how to test your code. It should be fun and useful. If any Agitators or other test afficionados are going to SD West, it would be great to see you at the talk, or afterwards as well.

Here are the details:

To Catch a Bug, You Have to Think Like a Bug

According to the rules:

2.1.3 A strike is made when a full setup of pins is knocked down with the first delivery in a frame.

After I completed the first story, I generated some characterization tests with JUnit Factory.

Before I move on to the next story, I want to revisit those tests and make sure we have not introduced any regressions.

Before I move on to spares and strikes, it would be nice to see how the score sheet looks. In an earlier post, I claimed that one of the reasons for integrating the UI early is to make sure the domain model will satisfy the requirements of the user interface. Let's see if it does.

My last post ended with this bold assertion:

If I am not mistaken, I have written enough code to pass the acceptance tests for this story.

It's at about this stage of the bowling example that people usually leap into a discussion about the appropriate data structure to store the rolls and the APIs for exposing the results.

In the previous installment, I wrote the code that implements rule 2.1.1. For rule 2.1.2, I finally start to add up some scores and show them in the score card.

It's an interesting word, 'test'. It can mean so many things. Before XP came along it used to mean

find out whether something works correctly

In my previous blog entry, I posted a set of acceptance tests for the first few stories. It's time to start writing the code to pass those tests. I prefer to discover the design through TDD rather than code directly to the customer-facing tests.

I have often written acceptance tests for code that has not yet been written (in fact, I wrote an article about it) but I have never written tests that will work with any number of implementations, each with their own architecture. I don't even know how to go about it, but that never stopped me before...

"scoring a game of bowling" is probably the most common application used when demoing TDD. It's so commonly known among the JUnit crowd that I chose one of Bob Martin's efforts as a demo for JUnit Factory.

The topic comes up about once a year on the TDD mailing list and it just came up again. By an odd coincidence, we just celebrated the completion of a new release of AgitarOne with a trip to Homestead Lanes, so I am all fired up about bowling despite my dismal performance (there was beer involved).

For me today was my best day so far at an EclipseCon, but as usual for a conference (except CITCON!) the most interesting stuff was what happened outside the talks...

We are proud to announce that registration is now open for the next Continuous Integration and Testing Conference, CITCON Dallas on April 27th & 28th. Space is limited to the first 100 registrants and attendance is free. Registration is on-line and while it is open until April 13th we do expect to fill all the available slots, so sign-up soon to reserve your spot.

The conference will be following the same Open Spaces (or unconference) format as the 2006 CITCON in Chicago and London. These prior CITCON drew enthusiastic practitioners at all levels of experience, all looking to share what they knew and to learn what they could. We're looking for more of the same in Dallas.

Please help spread the word about CITCON and we look forward to seeing you there!

Paul Julius
Jeffrey Fredrick

If you're at EclipseCon this week you might be interested in stopping by the panel Making Unit Testing Part of Your Development Process: How to Get Your Team to Do It. I'll be there as a panel member... but it should be a good panel anyway. ;-)

Mocks Aren't Stubs by Martin Fowler, is a very comprehensive look at two pairs of issues in testing: state-based verification vs behavior verification, and classical TDD vs Mockist TDD.

In light of the recent conversations about the adoption of developer testing on the junit list and Artima, this Turing Award lecture by Robert Floyd seems particularly appropriate. There's a particularly good quote where he is discussing a quote from Thomas Kuhn in "The Structure of Scientific Revolutions."

"Again from Kuhn:

"The older schools gradually disappear. In part their disappearance is caused by their members’ conversion to the new paradigm. But there are always some men who cling to one or another of the older views, and they are simply read out of the profession, which thereafter ignores their work."

In computing, there is no mechanism for reading such men out of the profession. I suspect they mainly become managers of software development. "

CITCON London registration will be closing this week, it is just hard to know if we will hit the deadline of Friday September 22nd or the cap of 120 people first! At last count we have 85 people signed up and if history is a guide the remaining spots will go fast.

For those who aren't aware CITCON is the Continuous Integration and Testing Conference, a free open spaces (or unconference) event on continuous integration and the testing that goes with it.

Given the open spaces format it is impossible to predict what the exact session topics will be but a sampling of the topics (and notes) from the CITCON Chicago event from earlier this year is available on the wiki. Also available are some photos, feedback, links to related blog entries and more... More than enough to be convinced that you should sign up today!

I just wrote a page on our internal wiki with our policy for dealing with build failures. We thought others might find it interesting so I am sharing it here (the links will be broken for obvious reasons).

Executive Summary

If the build fails, fix it.

In early August J.B. Rainsberger gave a webinar on TDD for J2EE:

Test-Driven Development is often introduced through simple examples, but many developers would rather dive into the deep end. This free webinar is for those people. J. B. Rainsberger, author of "JUnit Recipes," will show you architecture and design strategies to make it easier to "test-drive" J2EE components. You'll learn how to build a J2EE application while following the cardinal rule of Test-Driven Development: Never write a line of production code unless somewhere, a test has failed.

The recorded webinar is up on the Agitar website linked from this page. Registration is required but if you'd rather not register try out bugmenot.com.

There are many reasons why you might not want to fix a failing test right away. Maybe it's an acceptance test for a feature that you haven't written yet. Maybe it's a regression that it's just not practical to fix right now.

But what to do with that failing test?

If a little bit of feedback is good, a lot would be even better, right?

Is there such a thing as too much feedback?

XP doctrine says that you should stop tracking metrics once they have served their purpose. You should only have 3 or four "Things To Focus On". There's a reason for that.

It is critical to remember that Dos Equis Driven Design (XXD) is not about the beer. (I'm not saying there was no beer involved, but that isn't the point...)

I got fed up with updating the 20 step instructions on our wiki for configuring a new cruise control machine so I wrote a script and checked it in to CVS. Obvious really.

It's like what Whitehead said about notation: "By relieving the brain of all unnecessary work, a good notation sets it free to concentrate on more advanced problems, and in effect increases the mental power of the race."

Outstanding post, Brian. I always wondered what the little star was for in GMail. Now I know. Your post has a little gold one next to it.

Next week Agitar is hosting Ted Husted of the Struts development team (and iBATIS and MyFaces and Jakarta-Commons) in a webinar on What to Do if Your Code Has Few, If Any Tests?. I'm curious to hear what Ted has to say but his talk illustrates the kind of trade-off that exist as our company grows. One the one hand we can having interesting speakers and topics like this, but on the otherhand we're now scheduling the webinars for a global audience, so the scheduled times (7 am and 5 pm PDT) probably work better in just about every other timezone than this one. (</whine>)

Brian Marick says that tests are not specifications but I believe there is a more fundamental distinction.

This is the only thing that has ever worked for me.

1. Test your own code as well as you can
2. Next time someone checks in a slap-your-forehead bug, show them the test that would have caught the bug if only they had run the tests.
3. Show them how to run the tests themselves

It won't be long until everyone will want to have tests.

Most developers will not make the investment until they have seen proven returns. The management challenge is to find the early adopter with the courage and vision to take that first step.

Unlike Jeff, you won't catch me shamelessly plugging my EclipseCon talk on The Humble Plugin.

That suggests that a specification should not be written to a consistent level of precision. Precision is needed only where disputes have already occurred or are likely.

Just to make it easy for any potential candidates out there, here's my whole interview question for the J2EE position.

I want you to build a Hello World application in J2EE. I want you to do the simplest thing that can possibly work.

1. If you are interviewing for a company that makes tools for developer testing you should probably know something about developer testing.

I have these rules that I use for unit tests, primarily because I encounter so many teams that start writing end to end tests, call them unit tests, and give up because "testing takes too long". To me, a test is not a unit test if:

1. It talks to a database
2. Communicates across a network
3. Touches the file system
4. You can't run it at the same time as any of your other unit tests.
5. You have to do special things to your environment to run it (like editing configuration files).

Tests that do those things are okay, but to me they aren't unit tests, and they should be segregated from true unit tests.

On Thursday May 26 Jon Udell interviewed Alberto Savoia. Alberto had just been selected by InfoWorld as one of the 25 Top CTO's in the US in 2005. Jon asks Alberto about his vision for Agitator and Alberto demos Agitator extensively.

You can see the replay now if you don't mind registering. If you do, come back in a few days and we'll point you to an unguarded version.

Agitar has posted the videos from the November 17th Developer Testing Forum--they are now online at www.unikron.com/agitar1/. Presentations include Kent Beck, Sriram Sankar (Google), and an Expert Panel (Russell Gold, Oracle; Rob Mee, Pivotal; Sri Muthu, Wells Fargo; David Vydra, Testdriven.com).