Homepage ->  Individual Weblogs -> Kevin Lawrence  -> Test vs Spec or ForAll vs ThereExists

April 13, 2006 - Test vs Spec or ForAll vs ThereExists

Brian Marick says that tests are not specifications but I believe there is a more fundamental distinction.

It's often stated that language informs thinking, but I believe this is a case of a system of thought informing language.

I blogged a while back after a passionate company-wide discussion where it became apparent that we at Agitar were divided almost down the middle in our way of thinking about testing. The very fact that we had the debate and could put names to our differences allowed us to move the product forward in ways that would not have been possible before the debate.

The names we chose for the differences were For All and There Exists.

IMO the For All thinkers are more inclined to want to specify everything up front and then find the exceptions whereas There Exists people are more inclined to think about things one-at-a-time and only later go back and find the universal rule.

Perhaps the tendency, which Brian describes, to talk about tests as though they were specifications is an attempt to justify one-at-a-time thinking to the universalists ?


Posted by Kevin Lawrence at April 13, 2006 08:46 AM


Trackback Pings

TrackBack URL for this entry:
http://www.developertesting.com/mt/mt-tb.cgi/208

Listed below are links to weblogs that reference Test vs Spec or ForAll vs ThereExists:

» Tests, Specifications, Typing, Oh my! from John D. Mitchell's Blog

There's some interesting discussions taking place on the nature of tests. Brian Marick distinguishes between tests as specification vs. tests as examples. Michael Feathe...

[Read More]

Tracked on April 22, 2006 11:49 AM


Comments

I don't think this is a specification problem. It's a testing problem. Demonstrating that at least one member of a set is acceptable (i.e. one example of a function operating with a particular set of inputs on a particular platform at a particular time) is a far cry from demonstrating that all other members of the set are acceptable, but good testers are not aiming at either end of that spectrum. We strive instead to understand the set well enough to select and examine a sample that will provide the information our clients need.

The two elements are: understanding the set, and sampling the set. A single example may serve the purpose of explaining the whole set, but often that's not enough, especially if there are boundaries or error conditions that must also be handled. And a single example almost certainly does not provide a sufficient observation from which to draw an inference about the reliability of the software.

Let's set a different goal: "For sufficient", as in, "For sufficent X to support inferences of the kind we wish to make"

A good tester is one who, among other things, asks the questions and makes the investigations that shed light on what comprises a For Sufficient subset.

The argument in your company might be recast as a disagreement about what is sufficient. To have that conversation, you need to talk about risks and probabilities. It's not an all or nothing thing.

-- James

Posted by: james bach on April 14, 2006 01:32 PM

I like to alternate between finding a general rule and then finding specific examples that first demonstrate the general rule and then show where the rule does not hold and a new rule is required.

My hypothesis is that different people give different weight to each of these phases. These different modes are complementary but to realize the full benefit of the two modes of thinking, one has to recognize that the differences exist.

Kevin

Posted by: Kevin Lawrence [TypeKey Profile Page] on April 15, 2006 02:01 PM

Post a comment




Remember Me?