How to Design a Test Plan

Being a software tester means much more than just running through Acceptance Criteria on a story.  We need to think critically about every new feature and come up with as many ways as we can to test it.  When there are many permutations possible in a feature, we need to balance being thorough with testing in a reasonable amount of time.  Automation can help us test many permutations quickly, but too many people jump to automation without really thinking about what should be tested.

In this week's post, I'm going to describe a hypothetical feature and talk about how I would design a test plan for it.

The feature is called the Superball Sorter, and here is how it works:

• Superballs can be sorted among four children- Amy, Bob, Carol, and Doug
• The balls come in two sizes: large and small
• The balls come in six colors: red, orange, yellow, green, blue, and purple
• The children can be assigned one or more rules for sorting: for example, Amy could have a rule that says that she only accepts large balls, or Bob could have a rule that says he only accepts red or orange balls
• Distribution of the balls begins with Amy and then proceeds through the other children in alphabetical order, and continues in the same manner as if one was dealing a deck of cards
• Each time a new ball is sorted, distribution continues with the next child in the list
• The rules used must result in all the balls being sortable; if they do not, an error will be returned
• Your friendly developer has created a ball distribution engine that will create balls of various sizes and colors for you to use in testing

Here's a quick example:  let's say that Carol has a rule that she only accepts small balls.

The first ball presented for sorting is a large red ball.  Amy is first in the list, and she doesn't have any rules, so the red ball will go to her.

The next ball presented is a small blue ball.  Bob is second on the list, and he doesn't have any rules, so the blue ball will go to him.

The third ball is a large purple ball.  Carol is next on the list, BUT she has a rule that says that she only accepts small balls, so the ball will not go to her.  Instead the ball is presented to Doug, who doesn't have any rules, so the large purple ball will go to him.

So what we have after a first pass is:
Amy: large red ball
Bob: small blue ball
Carol: no ball
Doug: large purple ball

Since Doug got the most recent ball, we'd continue the sorting by offering a ball to Amy.

How should we test this?  Before I share my plan, you may want to take a moment and see what sort of test plan you would design yourself.  Then you can compare your plan with mine.

My test plan design philosophy always begins with testing the simplest possible option, and then gradually adding more complex scenarios.  So, I will begin with:

Part One:  No children have any rules

• If no children have any rules, we should see that the balls are always evenly distributed between Amy, Bob, Carol, and Doug in that order.  If we send in twenty balls, for example, we should see that each child winds up with five.  

Next,  I will move on to testing just one type of rule.  There are only two parameters for the size rule, but six parameters for the color rule, so I will start with the size rule:

Part Two: Size rules only

We could have anywhere from one child to four children with a rule.  We'll start with one child, and work up to four children.  Also, one child could have two rules, although that would be a bit silly, since the two rules would be accepting large balls only and accepting small balls only, which would be exactly the same as having no rules.  So let's write up some test cases:

A.  One child has a rule

• Amy has a rule that she only accepts large balls.  At the end of the test pass, she should only have large balls.
• Bob has a rule that he only accepts small balls.  At the end of the test pass, he should only have small balls.

B. Two children have rules

• Amy and Bob both have rules that they only accept large balls.  At the end of the test pass, they should only have large balls.
• Carol has a rule that she only accepts large balls, and Doug has a rule that he only accepts small balls.  At the end of the test pass, Carol should have only large balls, and Doug should have only small balls.

C. Three children have rules

• Amy, Bob, and Carol have rules that they only accept small balls. At the end of the test pass, they should only have small balls.
• Amy and Bob have rules that they only accept small balls, and Carol has a rule that she only accepts large balls.  At the end of the test pass, Amy and Bob should have only small balls, and Carol should have only large balls.
• Amy has a rule that she accepts both large balls and small balls, and Bob and Carol have rules that they only accept large balls.  At the end of the test pass, Amy should have both large and small balls (assuming that both were distributed during the test pass), and Bob and Carol should have only large balls.

D. Four children have rules

• Amy and Bob have rules that they only accept large balls, and Carol and Doug have rules that they only accept small balls.  
• All four children have a rule that they only accept large balls- this rule should return an error

Now that we have extensively tested the size rule, it's time to test the other rule in isolation:

Part Three: Color rules only

As with the size rule, anywhere from one to four children could have a color rule.  But this rule type is a bit more complex, because each child could have from one to six color rules.  Let's start simple with one child and one rule:

A. One child has one rule

• Bob accepts only red balls
• Bob accepts only orange balls
• Bob accepts only yellow balls
• Bob accepts only green balls
• Bob accepts only blue balls
• Bob accepts only purple balls

This tests that each color rule will work correctly on its own.  

B. One child has more than one rule

• Carol accepts only red and orange balls
• Carol accepts only red, orange, and yellow balls
• Carol accepts only red, orange, yellow, and green balls
• Carol accepts only red, orange, yellow, green, and blue balls
• Carol accepts only red, orange, yellow, green, blue and purple balls (which, again, is sort of silly, because it's like having no rule at all)

C. Two children have color rules

• Amy and Bob both accept only red balls
• Amy accepts only red balls and Bob accepts only blue balls
• Amy accepts only red and green balls and Bob accepts only blue and yellow balls
• Carol accepts only red, orange, and yellow balls, and Doug accepts only green balls

Note that there are MANY more possibilities here than what we are actually testing.  We are merely trying out a few different scenarios, such as one where one child has three rules and one child has one.

D. Three children have color rules

• Amy, Bob, and Carol accept only red balls
• Amy accepts only red balls, Bob accepts only orange balls, and Carol accepts only yellow balls
• Amy accepts only red balls, Bob accepts only red and orange balls, and Carol accepts only red, orange, and yellow balls

The last scenario above exercises a path where the children share one rule but not other rules.

E. Four children have color rules

• Amy, Bob, Carol, and Doug only accept purple balls- this should return an error
• Amy only accepts red and yellow balls, Bob only accepts orange balls, Carol only accepts yellow and blue balls, and Doug only accepts green balls- this should also return an error, because no one is accepting purple balls
• Amy only accepts red balls, Bob only accepts red and orange balls, Carol only accepts yellow balls, and Doug only accepts yellow, green, blue, and purple balls

Now that we've exercised both rule types separately, it's time to try testing them together!  Here's where things get really complicated.  Let's try to start simply with scenarios where each child has either a color rule or a size rule, but not both, and move on to more complex scenarios from there:

Part Four: Size and color rules

A. Children have one size rule or one color rule

• Doug only accepts large balls, and Bob only accepts red balls
• Doug only accepts large balls, Bob only accepts red balls, and Carol only accepts small balls
• Doug only accepts large balls, Bob only accepts red balls, and Carol only accepts small balls, and Amy only accepts yellow balls
• Amy and Doug only accept large balls, Bob only accepts small balls, and Carol only accepts purple balls
• Amy and Doug only accept large balls, and Bob and Carol only accept purple balls- this should return an error, because there's no one to accept any small balls other than the small purple balls

B. Children have both a size and a color rule

• Amy only accepts large red balls
• Amy only accepts large red balls, and Bob only accepts small blue balls
• Amy only accepts large red balls, Bob only accepts small blue balls, and Carol only accepts large green balls
• Amy only accepts large red balls, Bob only accepts small blue balls, Carol only accepts large green balls, and Doug only accepts small yellow balls- this should return an error

C. Children have a size rule and more than one color rule

• Amy only gets large red, orange, and yellow balls, Bob only gets small red, orange, and yellow balls, Carol only gets large green, blue, and purple balls, and Doug only gets small green, blue and purple balls
• Try the above scenario, but remove the large yellow ball from Amy's list- this should return an error, because there's no one to accept the large yellow balls

D. Children have more than one size rule and more than one color rule

• Amy: large red, large blue, small yellow; Bob: large orange, large purple, small green; Carol: large yellow, small red, small blue; Doug: large green, small orange, small purple
• Try the above scenario, but add a small purple ball to Amy's rules
• Try the first scenario, but change Doug's small purple rule to a large purple rule- this should return an error, because now there's no one to accept the small purple balls

And there you have it!  Forty-five rules that exercise a great many of the options and permutations that the Superball Sorter offers.  If you have read this far, you must enjoy testing as much as I do!  

How did my test plan compare to yours?  Did you think of things I didn't?  Be sure to let me know in the comments section!

You may have noticed that while this test plan is complete, it's not that easy to read, and it doesn't provide any way to keep track of test results.  Next week I'll talk about how to organize the test plan so you can run it quickly and record results easily.