Homepage ->  Individual Weblogs -> Kevin Lawrence  -> How Are Those Characterization Tests?

March 20, 2007 - How Are Those Characterization Tests?

Someone asked me how the characterization tests fared after such an extensive change. After all, I added new methods, new behavior to existing methods and I refactored extensively.

The tests all run - that's a good sign - and there are four failures:

When I delve a little deeper I find that they are all a variant on this test... 

      
      public void testAddBallThrowsIllegalArgumentException() throws Throwable {
        Frame frame = new Frame(null, false);
        frame.addBall(1);
        try {
          frame.addBall(10);
          fail("Expected IllegalArgumentException to be thrown");
        } catch (IllegalArgumentException ex) {
          assertEquals("ex.getMessage()", "Total pin count invalid", ex.getMessage());
          assertThrownBy(Frame.class, ex);
          assertEquals("frame.firstBall", 1, ((Number) getPrivateField(frame, "firstBall")).intValue());
          assertTrue("frame.needsMoreBalls()", frame.needsMoreBalls());
        }
      }

...and they failed because JUnit Factory is peeking into private fields. This kind of assertion is controversial (send your cards and letters to feedback@junitfactory.com) even within Agitar. Many of our customers love them because they let them test code that otherwise could not be tested but the love is not shared universally. The good news is that it is easy to regenerate the tests.

The new tests are even better than the old ones, like this set of tests for needsMoreBalls: 

      
      public void testNeedsMoreBalls() throws Throwable {
        Frame frame = new Frame(null, true);
        frame.addBall(0);
        frame.addBall(1);
        boolean result = frame.needsMoreBalls();
        assertFalse("result", result);
      }

      public void testNeedsMoreBalls1() throws Throwable {
        Frame frame = new Frame(new Frame(null, true), false);
        frame.addBall(10);
        frame.addBall(10);
        boolean result = frame.needsMoreBalls();
        assertTrue("result", result);
      }

      public void testNeedsMoreBalls2() throws Throwable {
        boolean result = new Frame(null, true).needsMoreBalls();
        assertTrue("result", result);
      }

      public void testNeedsMoreBalls3() throws Throwable {
        Frame frame = new Frame(null, true);
        frame.addBall(9);
        frame.addBall(1);
        boolean result = frame.needsMoreBalls();
        assertFalse("result", result);
      }

      public void testNeedsMoreBalls4() throws Throwable {
        Frame frame = new Frame(new Frame(new Frame(new Frame(null, true), false), false), true);
        frame.addBall(0);
        boolean result = frame.needsMoreBalls();
        assertTrue("result", result);
      }

      public void testNeedsMoreBalls5() throws Throwable {
        Frame frame = new Frame(new Frame(null, true), false);
        frame.addBall(10);
        frame.addBall(10);
        frame.addBall(1);
        boolean result = frame.needsMoreBalls();
        assertFalse("result", result);
      }

which, if I am not mistaken, covers both sides of every condition in needsMoreBalls().

The other tests are similarly detailed. If you want to see the tests for yourself, head on over to JUnit Factory and paste the Frame code into the text box in the demo.

It's important that we don't judge characterization tests by the same standards as we would judge other tests because they serve a different purpose. We, in the TDD community, often say that TDD is not about testing; the primary role of the test in TDD is to drive the design. In a similar way, characterization tests perform a very specific function - They tell us when something changed. Whether the previous behavior was correct or not, characterization tests tell you if it changed. We shouldn't expect characterization tests to look exactly like a human-written tests - no human would have nested Frames like that - but we should expect to be able to understand them when they fail.


Posted by Kevin Lawrence at March 20, 2007 06:04 PM


Trackback Pings

TrackBack URL for this entry:


Comments

Post a comment




Remember Me?