How to Log a Bug

Last week, we talked about all the things you should check before you log a bug, in order to make sure that what you are seeing is really a bug.  Once you have run through all your checks and you are sure you have a bug, it's time to log it.  But just throwing a few sentences in your team's bug-tracking software is not a good idea!  The way you log a bug and the details that you include can mean the difference between a bug being prioritized or left on the backlog; or the difference between a developer being able to find the problem, or closing the bug out with a "cannot repro" message.  In this post, I'll outline some best practices for logging a bug, with examples of what to do and what not to do.

Let's take an example bug from the hypothetical Superball Sorter that I described a few weeks ago. Imagine that when you tested the feature, you discovered that if three of the children have a rule where they accept only large balls of some color, the small purple ball is never sorted.

Here are the components of a well-logged bug:

Title: The title of the bug should begin with the area of the application it is referring to.  For example, if a bug was found in the Contacts section of your application, you could begin the title with "Contacts".  In this case, the area of the application is the Superball Sorter.  After the application area, you should continue the title with a concise statement of the problem.

RIGHT:  Superball Sorter: Small purple ball is not sorted when three children have large ball rules

WRONG: Small purple ball not sorted

While the second example gives at least a clue as to what the bug is about, it will be hard to find among dozens of other bugs later, when you might try to search by "Superball".  Moreover, it doesn't state what the conditions are when the ball is not sorted, so if there is another bug found later where this same ball isn't sorted, there could be confusion as to which bug is which.

Description: The first sentence of the bug should describe the issue in one sentence.  This sentence can provide a bit more detail than the title.  I often start this sentence with "When", as in "when I am testing x, then y happens".

RIGHT: When three children have sorting rules involving large balls, the small purple ball is not sorted.

WRONG: Doug doesn't get the small purple ball

There are a number of things wrong with this second example.  Firstly, the issue happens regardless of which three children have rules where they get only large balls, so referring to Doug here could be misleading.  Secondly, the statement doesn't describe what rules have been set up.  A developer could read this sentence and assume that the small purple ball is never sorted, regardless of what rules are set up.

Environment and browser details:  These can be optional if it's assumed that the issue was found in the QA environment and if the issue occurs on all browsers.  But if there's any chance that the developer won't know what environment you are referring to, be sure to include it.  And if the issue is found on one browser but not others, be sure to mention that detail.

RIGHT: This is happening in the Staging environment, on the Edge browser only

Steps to reproduce: The steps should include any configuration and login information, and clearly defined procedures to display the problem.

RIGHT:
1. Log in with the following user:
username: foobar
password: mebs47
2. Assign Amy a rule where she gets large red balls only
Assign Bob a rule where he gets large orange balls only
Assign Carol a rule where she gets large yellow balls only
3. Create a set of superballs to be sorted, and ensure that there is at least one small purple ball
4. Run the superball sorter
5. Check each child’s collection for the small purple ball or balls

WRONG:
Everyone has a rule but Doug, and no one is getting the small purple ball

The above example doesn't provide nearly enough information.  The developer won't know the login credentials for the user, and won't know that the three rules should be for large balls.

ALSO WRONG:
1. Open the application
2. Type “foobar” in the username field
3. Type “mebs47” in the password field
4. Click the login button
5. Go to the Superball Sorter rules page
6. Click on Amy’s name
7. Click on the large ball button
8. Click on the red ball button
9. Click the save button
10. Click on Bob’s name
etc. etc. etc.

These steps provide WAY too much information.  It can be safely assumed that the developer knows how to log in; simply providing the credentials should be enough.  It can also be safely assumed that the dev knows how to set a rule in the Superball Sorter, since he or she wrote the code.

Expected and Actual Result: For the sake of clarity, it's often a good idea to state what behavior you were expecting, and what behavior you got instead.  This can help prevent misunderstandings, and it's also helpful when a bug has been sitting on the backlog for months and you've forgotten how the feature is supposed to work.  

RIGHT:

Expected result: Doug gets the small purple ball, because he is the only child configured to accept it

Actual result: Doug does not get the small purple ball, and the ball remains unsorted

Screenshot or Stack Trace: Include this information only if it will be helpful.  

RIGHT: 

Exception: Ball was not recognized

Caused by: Ball.sort(Ball.java:11)

WRONG: 

Exception: An unknown error occurred 

ALSO WRONG:

The above screenshot is not particularly helpful.  While it shows that Doug doesn't have a small purple ball, that is easily conveyed by the Actual Result that has already been described; no picture is needed.

A clearly written bug is helpful to everyone: to the product owner, who prioritizes bug fixes; to the developer, who has to figure out what's wrong and fix it; and to the tester, who will need to retest the issue once it's fixed.  When you take care to log bugs well, it prevents frustration and saves everyone time.  

Before You Log That Bug...

Have you heard the ancient fable, "The Boy Who Cried Wolf"?  In the tale, a shepherd boy repeatedly tricks the people of his village by crying out that a wolf is about to eat his sheep.  The villagers run to his aid, only to find that it was a prank.  One day, the boy really does see a wolf.  He cries for help, but none of the villagers come because they are convinced that it must be another trick.

Similarly, we do not want to be "The Tester Who Cried Bug".  As software testers, if we repeatedly report bugs that are really user error, our developers won't believe us when we really find a bug.  To keep this from happening, let's take a look at the things we should check before we report a bug.  These tasks fall into two categories: checking for user error and gathering information for developers.

Check for User Error

• The first thing you should always check is to verify that the code you should be testing has actually been deployed.  When I was a new tester, I constantly made this mistake.  It's such a waste of time to keep investigating an issue with code when the problem is actually that the code isn't there!
• Are you testing in the right environment?  When you have multiple environments to test in, and they all look similar, it's easy to think you are testing in one environment when you are actually in another.  Take a quick peek at your URL if you are testing a web app, or at your build number if you are testing a mobile app.
• Do you understand the feature?  In a perfect world, we would all have great documentation and really well-written acceptance criteria.  In the real world, this often isn't the case!  Check with your developer and product owner to make sure that you understand exactly how the feature is supposed to behave.  Maybe you misunderstood something when you started to test.
• Have you configured the test correctly?  Maybe the feature only works when certain settings are enabled.  Think about what those settings are and go back and check them.  
• Are you testing with the right user?  Maybe this feature is only available to admin users or paid users.  Verify the criteria of the feature and check your user.
• Does the back-end data support the test?  Let's say you are testing that a customer's information is displayed.  You are expecting to see the customer's email address on the page, but the email is not there.  Maybe the problem is actually that the email address is null, and that is why it is not displaying.

If you have checked all of the above, and you are still seeing an issue, then it's time to think about reporting the bug.  But before you do, consider the questions the developer might ask you when he or she begins to investigate the issue.  It will save time for both of you if you have all of those questions answered ahead of time.

Information for the Developer

• Are you able to reproduce the issue?  You should be able to reproduce the issue at least once before logging the bug.  This doesn't mean that you shouldn't log intermittent issues, as they are important as well; but it does mean that you should have as much information as possible about when the issue occurs and when it doesn't.
• Do you have clear, reproducible steps to demonstrate the issue?  It is incredibly frustrating to a developer to hear that something is wrong with their software, but have only vague instructions available to use for investigation.  For best results, give the developer a specific user, complete with login credentials, and clear steps that they can use to reproduce the problem.
• Is this issue happening in Production?  Maybe this isn't a new bug; maybe the issue was happening already.  This is especially possible when you are testing old code that no one has looked at or used in a while.  See last week's post, The Power of Pretesting, for ideas on testing legacy software.
• Does the issue happen on every browser?  This information can be very helpful in narrowing down the possible cause of an issue.
• Does the issue happen with more than one user?  It's possible that the user you are testing with has some kind of weird edge case in their configuration or their data.  This doesn't mean that the issue you are seeing isn't a bug; but if you can show that there are some users where the issue is not happening, it will help narrow the scope of the problem.
• Does the issue happen if the data is different?  Try varying the data and see if the issue goes away.  Maybe the problem is caused by a data point that is larger than the UI is expecting, or a field that is missing a value.  The more narrowly you can pinpoint the problem, the faster the developer can fix it.

The ideal relationship between a tester and a developer is one of mutual trust.  If you make sure to investigate each issue carefully before reporting it, and if you are able to report issues with lots of helpful details, your developer will trust that when you cry "Bug", it's something worth investigating!

The Power of Pretesting

Having been in the software testing business for a few years now, I've become accustomed to various types of testing: Acceptance Testing, Regression Testing, Exploratory Testing, Smoke Testing, etc.  But in the last few weeks, I've been introduced to a type of testing I hadn't thought of before: Pretesting.

On our team, we are working to switch some automatically delivered emails from an old system to a new system.  When we first started testing, we were mainly focused on whether the emails in the new system were being delivered.  It didn't occur to us to look at the content of the new emails until later, and then we realized we had never really looked at the old emails.  Moreover, because the emails contained a lot of detail, we found that we kept missing things: some extra text here, a missing date there.  We discovered that the best way to prevent these mistakes was to start testing before the new feature was delivered, and thus, Pretesting was born.

Now whenever an email is about to be converted to the new system, we first test it in the old system.  We take screenshots, and we document any needed configuration or unusual behavior.  Then when the email is ready for the new system, it's easy to go back and compare with what we had before.  This is now a valuable tool in our testing arsenal, and we've used it in a number of other areas of our application.

When should you use Pretesting?

• when you are testing a feature you have never tested before
• when no one in your company seems to know how a feature works
• when you suspect that a feature is currently buggy
• when you are revamping an existing feature
• when you are testing a feature that has a lot of detail

Why should you Pretest?

Pretesting will save you the headache of trying to remember how something "used to work" or "used to look".  If customers will notice the change you are about to make, it's good to make note of how extensive the change will be.  If there are bugs in the existing feature, it would be helpful to know that before the development work starts, because the developer could make those bug fixes while in the code.  Pretesting is also helpful for documenting how the feature works, so you can share those details with others who might be working on or testing the feature.

How to Pretest:

1. Conduct exploratory testing on the old feature.  Figure out what the Happy Path is.
2. Document how the Happy Path works and include any necessary configuration steps
3. Continue to test, exploring the boundaries and edge cases of the feature
4. Document any "gotchas" you may find- these are not bugs, but are instead areas of that feature that might not work as someone would expect
5. Log any bugs you find and discuss the with your team to determine whether they should be fixed with the new feature or left as is
6. Take screenshots of any complicated screens, such as emails, messages, or screens with a lot of text, images, or buttons. Save these screenshots in an easily accessible place such as a wiki page.

When the New Feature is Ready:

1. Run through the same Happy Path scenario with the new feature, and verify that it behaves the same way as the old feature
2. Test the boundaries and edge cases of the feature, and verify that it behaves in the same way as the old feature
3. Verify that any bugs the team has decided to fix have indeed been fixed
4. Compare the screenshots of the old feature with the screenshots of the new feature, and verify that they are the same (with the exceptions of anything the team agreed to change)
5. Do any additional testing needed, such as testing new functionality that the old feature didn't have or verifying that the new feature integrates with other parts of the application

The power of Pretesting is that it helps you notice details you might otherwise miss in a new feature, and as a bonus, find existing bugs in the old feature.  Moreover, testing the new feature will be easy, because you will have already created a test plan.  Your work will help the developer do a better job, and your end users will appreciate it!

Automating Tests for a Complicated Feature

Two weeks ago, I introduced a hypothetical software feature called the Superball Sorter, which would sort out different colors and sizes of Superballs among four children.  I discussed how to create a test plan for the feature, and then last week I followed up with a post about how to organize a test plan using a simple spreadsheet.  This week I'll be describing how to automate tests for the feature.

In case you don't have time to view the post where I introduced the feature, here's how it works:

• Superballs can be sorted among four children- Amy, Bob, Carol, and Doug
• The balls come in two sizes: large and small
• The balls come in six colors: red, orange, yellow, green, blue, and purple
• The children can be assigned one or more rules for sorting: for example, Amy could have a rule that says that she only accepts large balls, or Bob could have a rule that says he only accepts red or orange balls
• Distribution of the balls begins with Amy and then proceeds through the other children in alphabetical order, and continues in the same manner as if one was dealing a deck of cards
• Each time a new ball is sorted, distribution continues with the next child in the list
• The rules used must result in all the balls being sortable; if they do not, an error will be returned
• Your friendly developer has created a ball distribution engine that will create balls of various sizes and colors for you to use in testing

When automating tests, we want to keep our tests as simple as possible.  This means sharing code whenever we can.  In examining the manual test plan, we can see that there are three types of tests here:

• A test where none of the children have any rules
• Tests where all the children have rules, but the rules result in some balls not being sortable
• Tests where one or more children have rules, and all of the balls are sortable

We can create a separate test class for each of these types, which will make it easy for us to share code inside each class.  We'll also have methods that will be shared among the classes:

child.deleteBalls()- this will clear all the distributed balls, getting ready for the next test

child.deleteRules()- this will clear out all the existing rules, getting ready for the next test

distributeBalls(numberOfBalls)- this will randomly generate a set number of balls of various sizes and colors and distribute them one at a time to the children, according to the rules

verifyEvenDistribution(numberOfBalls)- this is for scenarios where none of the children have rules; it will take the number of balls distributed and verify that each child has one-fourth of the balls

child.addRule(Size size, Color color)- this will set a rule for a child; each child can have more than one rule, and either the size or color (but not both) can be null

child.verifyRulesRespected()- for the specified child, this will iterate through each ball and each rule and verify that each ball has respected each rule

child.addFourthRuleAndVerifyError(Size size, Color color)- for the tests in the InvalidRules class, it will always be the fourth child's rules that will trigger the error, because it's only with the fourth child that the Sorter realizes that there will be balls that can't be sorted.  So this method will assert that an error is returned.

Each test class should have setup and cleanup steps to avoid repetitive code.  However, I would use child.deleteBalls() and child.deleteRules() for each child in both my setup and cleanup steps, just in case there is there is an error in a test that causes the cleanup step to be missed.

For the DistributionWithNoRules test class, each test would consist of:

distributeBalls(numberOfBalls);
verifyEvenDistribution(numberOfBalls);

All that would vary in each test would be the number of balls passed in.

For the DistributionWithRules test class, each test would consist of:

child.AddRule(Size size, Color color); --repeated as many times as needed for each child
distributeBalls(numberOfBalls);
child.verifyRulesRespected(); --repeated for each child that has one or more rules

Finally, for the InvalidRules test class, each test would consist of:

child.AddRule(Size size, Color color); --repeated as many times as needed for the first three children
child.addFourthRuleAndVerifyError(Size size, Color color); --verifying that the fourth rule triggers an error

The nice thing about organizing the tests like this is that it will be easy to vary the tests.  For example, you could have the DistributionWithNoRules test class run with 40, 400, and 800 balls; or you could set up a random number generator that would generate any multiple of four for each test.

You could also set up your DistributionWithRules and InvalidRules test classes to take in rule settings from a separate table, varying the table occasionally for greater test coverage.

Astute readers may have noticed that there are a few holes in my test plan:

• how would I assert even distribution in a test scenario where no children have rules and the number of balls is not evenly divisible by four?
• how can I show that a child who doesn't have a rule is still getting the correct number of balls, when there are one or more children with a rule?  For example, if Amy has a rule that she only gets red balls, and Bob, Carol, and Doug have no rules, how can I prove that Bob, Carol and Doug get an even distribution of balls?
• will the Superball Sorter work with more or less than four children? How would I adjust my test plan for this?

I'll leave it to you to think about handling these things, and perhaps I will tackle them in future posts.

I've had a lot of fun over the last few weeks creating the idea of the Superball Sorter and thinking of ways to test it!  I've started to actually write code for this, and someday when it's finished I will share it with you.