What to Put in a Smoke Test

The term "smoke test" is usually used to describe a suite of basic tests that verify that all the major features of an application are working.  Some use the smoke test to determine whether a build is stable and ready for further testing.  I usually use a smoke test as the final check in a deploy to production.  In today's post, I'll share a cautionary tale about what can happen if you don't have a smoke test.  Then I'll continue that tale and talk about how smoke tests can go wrong.

Early in my testing career, I worked for a company that had a large suite of manual regression tests, but no smoke test.  Each software release was difficult, because it was impossible to run all the regression tests in a timely fashion.  With each release, we picked which tests we thought would be most relevant to the software changes and executed those tests.

One day, in between releases, we heard that there had been a customer complaint that our Global Search feature wasn't working.  We investigated and found that the customer was correct.  We investigated further and discovered that the feature hadn't worked in weeks, and none of us had noticed.  This was quite embarrassing for our QA team!

To make sure that this kind of embarrassment never happened again, one of our senior QA engineers created a smoke test to run whenever there was a release to production.  It included all the major features, and could be run fairly quickly.  We felt a lot better about our releases after that.

However, the tester who created the test kept adding test steps to the Smoke Test.  Every time a new feature was created, a step was added to the smoke test.  If we found a new bug in a feature, even it was a small one, a step checking for the bug was added to the smoke test.  As the months went on, the smoke test took longer and longer to execute and became more and more complicated.  Eventually the smoke test itself took so much time that we didn't have time to run our other regression tests.

Clearly there needs to be a happy medium between having no smoke test at all, and having one that takes so long to run that it's no longer a smoke test.  In order to decide what goes in a smoke test, I suggest asking these three questions:

1. What would absolutely embarrass us if it were broken in this application?

Let's use an example of an e-commerce website to consider this question.  For this type of website, it would be embarrassing or even catastrophic if a customer couldn't:

• search for an item they were looking for
• add an item to their cart
• log in to their account
• edit their information

So at the very least, a smoke test for this site should include a test for each of these features.

2. Is this a main feature of the application?

Examples of features in an e-commerce website that would be main features, but less crucial ones, might be:

• wish list functionality
• product reviews
• recommendations for the user

If these features were broken, it wouldn't be catastrophic, but they are features that customers expect.  So a test for each one should be added.

3. If there was a bug here, would it stop the application from functioning?

No one wants to have bugs in their application!  But some bugs are more important than others.  If the e-commerce website had an issue where their "Add to Cart" button was off-center, it might look funny, but it wouldn't stop customers from shopping.  

But a bug where a customer couldn't remove an item from their cart might keep them from checking out with the items they want, which would affect sales.  So a test to check that items can be removed from a cart would be important in a Smoke Test.

With these questions in mind, here is an example of a smoke test that could be created for an e-commerce site:

1. Log in

2. Verify product recommendations are present

3. Do a search for a product

4. Read a review of a product

5. Add an item to the cart

6. Add a second item to the cart and then delete it

7. Edit customer information

8. Check out

9. Write a review

A smoke test like this wouldn't take very long to execute manually, and it would also be easy to automate.  

Whenever new features are added to the application, you should ask yourself the first two questions to determine whether a test for the feature should be added to the smoke test.  And whenever a bug is found in the product, you should ask yourself the third question to determine whether a test for that issue should be added to the smoke test.

Because we want our applications to be of high quality, it's easy to fall into the trap of wanting to test everything, all the time.  But that can create a test burden that keeps us so busy that we don't have time for anything else.  Creating a simple, reliable smoke test can free us up for other activities, such as doing exploratory testing on new features or creating nightly automated tests.