Six Steps to Writing an Effective Test Report

As testers, we know how important it is to test our software thoroughly and document our findings meticulously.  But all of our talent will be useless if we can't effectively communicate our test results to others!  If your test results are written in a giant, poorly organized spreadsheet with tiny text and lots of unnecessary details, even the most dedicated test manager will find her eyes glazing over with boredom when she looks at it.  In this post, I'll describe six steps to take to make sure that you can communicate your findings to others efficiently and effectively.

Step One: Determine what goal you are trying to accomplish with your report

Why are you creating your report?  What do you want people to understand when they read it?  You might be creating a report to demonstrate that you have tested all the acceptance criteria in the user story.  You could be showing your manager exactly which areas of the application are not functioning properly.  You could be demonstrating that after a recent code change several bugs were fixed.  

There are all kinds of reasons to create a report, but if you don't stop and think about why you are creating it, it's probably not going to be very clear to your readers.  Simply sending over your test notes is not enough to communicate effectively.

Step Two: Focus on the viewer's needs

Who will be reading your report?  Will it be a developer, your test manager, your product owner, your team lead, or the CTO of your company?  You will want to tailor your test report for your audience.  The CTO of your company is probably very busy and will not care how many permutations of your form data you ran, or how you developed the permutations.  She will want to see that you ran 100 tests and that 99 passed.  Your product owner will want to see that you have tested the user stories and that the outcomes were as expected.  Your test manager might be interested in how many different permutations of the test you ran, and what logic you used to create them.  Your developer might want to know what test data you used and what your server response times were.  

You can see how the interests of your reader will vary quite a bit depending on their role, so think about how you can best present the information that he or she wants.

For steps three through six, we'll once again use the example of the Superball Sorter.  This is a hypothetical software feature that sorts out superballs of varying sizes and colors to four children according to a set of rules.  

Step Three: Avoid Extraneous Information

Make sure that your report contains only the information that your reader needs.  Extraneous information means that a reader has to sift through your report to find the important results.  Consider this report, which shows the results of two tests where two children have a sorting rule:

Rules	Amy	Bob	Carol	Doug
Amy-small blue balls; Doug- large green balls	small blue ball; small blue ball; small blue ball; small blue ball	large red ball; small orange ball; large yellow ball; small purple ball	large purple ball; small green ball; large yellow ball; small red ball	large green ball; large green ball; large green ball; large green ball
Bob- large red balls; Carol- small yellow balls	large orange ball; small purple ball; large yellow ball; small green ball	large red ball; large red ball; large red ball; large red ball; large red ball	small yellow ball; small yellow ball; small yellow ball; small yellow ball	small blue ball; small green ball; large purple ball; small orange ball

Did the tests pass?  How long did it take for you to determine that by looking at the test report?  There is a lot of information here that is unnecessary.  It doesn't matter that Bob got the large red ball first and the small orange ball second.  What matters is that Amy only got small blue balls and Doug got large green balls.  Contrast that test report with this one:

Rules	Rules respected?
Amy-small blue balls; Doug- large green balls	Yes
Bob- large red balls; Carol- small yellow balls	Yes

Here you can very quickly see what rules were set, and whether those rules were respected.  If a reader of your report needs to know what balls Bob got, they can ask you for those details and you can look them up in your notes.

Step Four: Make the report visually immediate

We are all busy people; developing and testing software is fast-paced and time-consuming!  Your manager or CTO is probably sent dozens of reports and emails a day, so make your report so easy to read that just a glance at it will give them information.  Take a look at this test report:

Number of Rules	Pass/Fail
0
1
2
3
4

How many seconds does it take you to see that a test failed?  This report is pretty immediate!  It's also really easy to see how many rules were used when the test failed.  Compare that with this test report, which shows the same tests and the same results:

Test Case	Result
None of the children have rules	The balls are sorted evenly amongst the children
One child has a rule	The child’s rule is respected
Two children have rules	The two children’s rules are respected
Three children have rules	The three children’s rules are respected
Four children have rules	None of the balls are sorted

A reader needs to read through all the text of this report in order to see that the fourth test failed.

Step Five: Make the report easy to read

In addition to making the report visually immediate, it also needs to be easy to read. Take a look at this example, where two tests are run where three children are given rules:

Rules	Rules respected?
Amy- small blue; Bob- large blue; Carol- small purple	Amy gets only small blue balls, and Bob gets only large blue balls, but Carol gets balls other than the small purple balls
Amy- large blue; Bob- small purple; Carol- small yellow	Amy gets only large blue balls, Bob gets only small purple balls, and Carol gets only small yellow balls

This report shows very quickly that one of the tests failed, but in order to see why the test failed, it's necessary to read through the whole description to see that Carol's rule was not respected.  This report conveys exactly the same information:

Rules	Amy	Bob	Carol
Amy- small blue; Bob- large blue; Carol- small purple	PASS	PASS	FAIL
Amy- large blue; Bob- small purple; Carol- small yellow	PASS	PASS	PASS

With this report, it is so easy to see that Carol's rule was not respected, and it's also easy to look to the left to see which rule she had.

Step Six: Make the report readable without any additional explanation

How long does it take for you to figure out what this report means? 

Rules	Result
A-SB; B-LO; C-L; D-S	A-Y; B-Y; C-Y; D-N
A-L; B-S; C-Y; D-P	A-Y; B-N; C-Y; D-Y
A-LY; B-L; C-S; D-SG	A-Y; B-Y; C-N; D-Y

It's fine to use all sorts of abbreviations when you are testing and taking notes for yourself, but you shouldn't need to provide a key to your reader in order to have them interpret it.  Who but the most interested testers are going to take the time to see where the bug is here?

This report conveys exactly the same information:

Test One	Amy- small blue	Bob- large orange	Carol- large	Doug- small
Rule respected?	Yes	Yes	Yes	No
Test Two	Amy- large	Bob- small	Carol- yellow	Doug- purple
Rule respected?	Yes	No	Yes	Yes
Test Three	Amy- large yellow	Bob- large	Carol- small	Doug- small green
Rule respected?	Yes	Yes	No	Yes

It's easy to see exactly what rules each child was given for each test.  Through the use of color, the report demonstrates very clearly where the bug is: whenever a child is given a rule that they should get only small balls, that rule is not respected.

Conclusion:

In today's fast-paced world, we all have vast amounts of information coming to us every day.  If we are going to make a difference with our testing and influence decision-making where we work, we need to be able to convey our test results in ways that clearly show what is going on with our software and what should be done to improve it.