Rethinking the Pyramid: The Automation Test Wheel

Anyone who has spent time working on test automation has likely heard of the Test Automation Pyramid.  The pyramid is typically made of three horizontal sections: UI Tests, API Tests, and Unit Tests.  The bottom section is the widest section, and is for the unit tests.  The idea is that there should be more unit tests run than any other kind of tests.  The middle section is for the API tests, and the idea is that fewer API tests should be run than unit tests.  Finally, the top section is for the UI tests, and the idea is that the least number of tests run should be UI tests, because they take the most time and are the most fragile.

In my opinion there are two things wrong with the pyramid: it leaves out many types of automated tests, and it assumes that the number of tests is the best indicator of appropriate test coverage.  I propose a new way of thinking about automated testing:  the Automation Test Wheel.

Each of these test types can be considered as spokes in a wheel; none is more important than another, and they are all necessary.  The size of each section of the wheel does not indicate the quantity of the tests to be automated; each test type should have the number of tests that are needed in order to verify quality in that area.  Let's take a look at each test type.

Unit Tests:  A unit test is the smallest automated test possible.  It tests the behavior of just one function or method.  For example, if I had a method that tested whether a number was zero, I could write these unit tests:

• A test that passes a zero to the method and validates that it is identified as a zero
• A test that passes a one to the method and validates that it is identified as non-zero
• A test that passes a string to the method and validates that the appropriate exception is thrown

Because unit tests are independent of all other services and because they run so quickly, they are a very effective way of testing code.  They are often written by the developer who wrote the method or function, but they can also be written by others.  Each method or function should have at least one unit test associated with it.

Component Tests: These tests check the various services that the code is dependent on.  For example, if we had code that called the GitHub API, we could write a component test that would make a call to the API and verify that the API was running.  Other examples of component tests are pinging a server or making a call to a database and verifying that a response was received.  There should be at least one component test for each service the code relies on.

Services Tests:  These are tests that check the web services that are used in our code.  In today's applications, web services often use API requests.  For example, if we have an API with POST, GET, PUT, and DELETE requests, we will want to have automated tests that check each request type.  We will want to have both "happy path" tests that check that a valid request returns an appropriate response, and also negative tests that verify that an invalid request returns an appropriate error code.

User Interface (UI) Tests: UI tests verify that end-user activities work correctly.  These are the tests that will fill out text fields and click on buttons.  As a general rule, anything that can be tested with a unit, component, or service test should be tested by those methods instead.  UI tests should focus solely on the user interface.

Visual Tests: Visual tests verify that elements are actually appearing on the screen.  This is slightly different from UI tests, because the UI tests are focusing on the functionality of the user interface rather than the appearance.  Examples of visual tests would be: verifying that a button's label is rendered correctly, and verifying that the correct product image is appearing on the screen.

Security Tests:  These are tests that verify that security rules are being respected.  These tests can overlap with services tests, but should still be considered separately.  For example, a security test could check to make sure that an authorization token cannot be generated with an invalid username and password combination.  Another security test would be to make a GET request with an authorization token for a user who should not have access to that resource, and verify that a 403 response is returned.

Performance Tests: Automated performance tests can verify that request response times happen within an appropriate time period.  For example, if your company has decided that GET requests should never take longer than two seconds, test requests can be set to return a failure state if the request takes longer than that time.  Web page load times can also be measured with performance tests.

Accessibility Tests:  Automated accessibility tests can check a variety of things.  When combined with UI tests, they can verify that images have text descriptions for the visually impaired.  Visual tests can be used to verify that the text on the screen is the correct size. 

You may have noticed that the above descriptions often overlap each other.  For example, security tests might be run through API testing, and visual tests might be run through UI testing.  What is important here is that each area is tested thoroughly, efficiently, and accurately.  If there is a spoke missing from the wheel, you will never be comfortable relying on your automation when you are doing continuous deployment. 

Next week, I'll discuss how we can fit all these tests into a real-world application testing scenario!

Fifteen Free Tools to Help With Testing

There are a great many articles, blog posts, and presentations that discuss automation frameworks and strategies.  But even the most robust automation framework won't eliminate the need to do exploratory testing.  There will always be situations where we need to generate a large amount of text to test a text field or where we need to encode a string in HTML to test for cross-site scripting.  In this week's post, I share fifteen of my favorite free tools that make testing faster and easier.  

Text Tools:

1. Letter Count:  This tool will count the characters or words in a block of text.  I use it for creating strings with a specific character count when I test text fields.

2. Lorem Ipsum Generator: I use this tool when I need to generate large amounts of text for text fields where a user will be able to enter several paragraphs of text.

3. Convert Case: This tool comes in handy when I'm testing with Postman and my assertions are expecting the exact casing for string comparison.  Convert Case will set all the characters in a string to lower case, upper case, sentence case, alternating case, and more.

JSON Tools:

4. Pretty Print: JSON objects need indentation to be easily readable.  This tool will take care of all of the indentation and spacing for you.  This is especially helpful when you receive flattened JSON in a response and you want to be able to read through it.

5. Online JSON Viewer: This tool will flatten your JSON for you by removing all the white spaces, which is helpful when you need to use your JSON in code.

6. JSON Lint: I use this tool whenever I'm using JSON in a test and getting 400 errors.  I paste the JSON into the tool and it will tell me whether my JSON is valid.  If it's not valid, it shows me the line with the error.

Encoding and Decoding Tools:

7. URL Encoder/Decoder: This tool is great for testing cross-site scripting.  Simply paste in your script and it will encode it in valid HTML format for you.

8. JWT.IO: This official JWT documentation includes a tool that will decode a JWT.  It's really helpful for testing authorization, because you can see exactly what information your JWT is sending.  

9. Base 64 Decode and Encode: When you have an image or string that's been encoded into Base 64, this is an easy way to decode it.  

GUID Tools:

10. GUID Generator: I often need random GUIDs for my testing, and this tool will generate as many as I need.

11. GUID Test and Validate: I use this tool to check GUIDs to make sure that they are valid. 

Miscellaneous Tools:

12. Rubular: This is a regex editor that allows you to put in a regex, add a string to test, and find out whether your string matches the regex.  It is specifically designed for Ruby but can be used with other languages as well.  

13. NumVerify: This tool is helpful when you are testing international phone numbers.  It lets you know whether a given number is valid, and which country it is valid in.  

14. WePay Testing: I use this site when I need a fake bank account to test with.  It also lists fake credit card numbers for use in testing.  Some of the features may be specific to Chase Bank.  

15. Online Random File Generator: When you need to test files of a very specific size, this tool uses random strings to generate a file with exactly the size you need.

Just as we use tools in our daily life to make our tasks easier, these free online tools can help speed up our testing, giving us more time to write test automation or do further exploratory testing.  

What are your favorite free tools for testing?  Let me know in the comments below!  

Six Steps to Writing an Effective Test Report

As testers, we know how important it is to test our software thoroughly and document our findings meticulously.  But all of our talent will be useless if we can't effectively communicate our test results to others!  If your test results are written in a giant, poorly organized spreadsheet with tiny text and lots of unnecessary details, even the most dedicated test manager will find her eyes glazing over with boredom when she looks at it.  In this post, I'll describe six steps to take to make sure that you can communicate your findings to others efficiently and effectively.

Step One: Determine what goal you are trying to accomplish with your report

Why are you creating your report?  What do you want people to understand when they read it?  You might be creating a report to demonstrate that you have tested all the acceptance criteria in the user story.  You could be showing your manager exactly which areas of the application are not functioning properly.  You could be demonstrating that after a recent code change several bugs were fixed.  

There are all kinds of reasons to create a report, but if you don't stop and think about why you are creating it, it's probably not going to be very clear to your readers.  Simply sending over your test notes is not enough to communicate effectively.

Step Two: Focus on the viewer's needs

Who will be reading your report?  Will it be a developer, your test manager, your product owner, your team lead, or the CTO of your company?  You will want to tailor your test report for your audience.  The CTO of your company is probably very busy and will not care how many permutations of your form data you ran, or how you developed the permutations.  She will want to see that you ran 100 tests and that 99 passed.  Your product owner will want to see that you have tested the user stories and that the outcomes were as expected.  Your test manager might be interested in how many different permutations of the test you ran, and what logic you used to create them.  Your developer might want to know what test data you used and what your server response times were.  

You can see how the interests of your reader will vary quite a bit depending on their role, so think about how you can best present the information that he or she wants.

For steps three through six, we'll once again use the example of the Superball Sorter.  This is a hypothetical software feature that sorts out superballs of varying sizes and colors to four children according to a set of rules.  

Step Three: Avoid Extraneous Information

Make sure that your report contains only the information that your reader needs.  Extraneous information means that a reader has to sift through your report to find the important results.  Consider this report, which shows the results of two tests where two children have a sorting rule:

Rules	Amy	Bob	Carol	Doug
Amy-small blue balls; Doug- large green balls	small blue ball; small blue ball; small blue ball; small blue ball	large red ball; small orange ball; large yellow ball; small purple ball	large purple ball; small green ball; large yellow ball; small red ball	large green ball; large green ball; large green ball; large green ball
Bob- large red balls; Carol- small yellow balls	large orange ball; small purple ball; large yellow ball; small green ball	large red ball; large red ball; large red ball; large red ball; large red ball	small yellow ball; small yellow ball; small yellow ball; small yellow ball	small blue ball; small green ball; large purple ball; small orange ball

Did the tests pass?  How long did it take for you to determine that by looking at the test report?  There is a lot of information here that is unnecessary.  It doesn't matter that Bob got the large red ball first and the small orange ball second.  What matters is that Amy only got small blue balls and Doug got large green balls.  Contrast that test report with this one:

Rules	Rules respected?
Amy-small blue balls; Doug- large green balls	Yes
Bob- large red balls; Carol- small yellow balls	Yes

Here you can very quickly see what rules were set, and whether those rules were respected.  If a reader of your report needs to know what balls Bob got, they can ask you for those details and you can look them up in your notes.

Step Four: Make the report visually immediate

We are all busy people; developing and testing software is fast-paced and time-consuming!  Your manager or CTO is probably sent dozens of reports and emails a day, so make your report so easy to read that just a glance at it will give them information.  Take a look at this test report:

Number of Rules	Pass/Fail
0
1
2
3
4

How many seconds does it take you to see that a test failed?  This report is pretty immediate!  It's also really easy to see how many rules were used when the test failed.  Compare that with this test report, which shows the same tests and the same results:

Test Case	Result
None of the children have rules	The balls are sorted evenly amongst the children
One child has a rule	The child’s rule is respected
Two children have rules	The two children’s rules are respected
Three children have rules	The three children’s rules are respected
Four children have rules	None of the balls are sorted

A reader needs to read through all the text of this report in order to see that the fourth test failed.

Step Five: Make the report easy to read

In addition to making the report visually immediate, it also needs to be easy to read. Take a look at this example, where two tests are run where three children are given rules:

Rules	Rules respected?
Amy- small blue; Bob- large blue; Carol- small purple	Amy gets only small blue balls, and Bob gets only large blue balls, but Carol gets balls other than the small purple balls
Amy- large blue; Bob- small purple; Carol- small yellow	Amy gets only large blue balls, Bob gets only small purple balls, and Carol gets only small yellow balls

This report shows very quickly that one of the tests failed, but in order to see why the test failed, it's necessary to read through the whole description to see that Carol's rule was not respected.  This report conveys exactly the same information:

Rules	Amy	Bob	Carol
Amy- small blue; Bob- large blue; Carol- small purple	PASS	PASS	FAIL
Amy- large blue; Bob- small purple; Carol- small yellow	PASS	PASS	PASS

With this report, it is so easy to see that Carol's rule was not respected, and it's also easy to look to the left to see which rule she had.

Step Six: Make the report readable without any additional explanation

How long does it take for you to figure out what this report means? 

Rules	Result
A-SB; B-LO; C-L; D-S	A-Y; B-Y; C-Y; D-N
A-L; B-S; C-Y; D-P	A-Y; B-N; C-Y; D-Y
A-LY; B-L; C-S; D-SG	A-Y; B-Y; C-N; D-Y

It's fine to use all sorts of abbreviations when you are testing and taking notes for yourself, but you shouldn't need to provide a key to your reader in order to have them interpret it.  Who but the most interested testers are going to take the time to see where the bug is here?

This report conveys exactly the same information:

Test One	Amy- small blue	Bob- large orange	Carol- large	Doug- small
Rule respected?	Yes	Yes	Yes	No
Test Two	Amy- large	Bob- small	Carol- yellow	Doug- purple
Rule respected?	Yes	No	Yes	Yes
Test Three	Amy- large yellow	Bob- large	Carol- small	Doug- small green
Rule respected?	Yes	Yes	No	Yes

It's easy to see exactly what rules each child was given for each test.  Through the use of color, the report demonstrates very clearly where the bug is: whenever a child is given a rule that they should get only small balls, that rule is not respected.

Conclusion:

In today's fast-paced world, we all have vast amounts of information coming to us every day.  If we are going to make a difference with our testing and influence decision-making where we work, we need to be able to convey our test results in ways that clearly show what is going on with our software and what should be done to improve it.

How to Reproduce a Bug

Have you ever seen something wrong in your application, but you haven't been able to reproduce it?  Has a customer ever reported a bug with a scenario that you just couldn't recreate?  It is tempting to just forget about these bugs, but chances are if one person has seen the issue, other people will see it as well.  In this post I'll discuss some helpful hints for reproducing bugs and getting to the root cause of issues.

Gather Information

The first thing to do when you have a bug to reproduce is to gather as much information as you can about the circumstances of the issue.  If it's a bug that you just noticed, think about the steps that you took before the bug appeared.  If it's a bug that someone else has reported, find out what they remember about the steps they took, and ask for details such as their operating system, browser, and browser version.

One Step at a Time

Next, see if you can follow the steps that you or the reporter of the bug took.  If you can reproduce the bug right away, you're in luck!  If not, try making one change at a time to your steps, and see if the bug will appear.  Don't just thrash around trying to reproduce the issue quickly; if you're making lots of disorganized changes, you might reproduce the bug and not know how you did it.  Keep track of the changes you made so you know what you've tried and what you haven't tried yet.

Logs and Developer Tools

Application logs and browser developer tools can be helpful in providing clues to what is going on behind the scenes in an application.  A browser's developer tools can generally be accessed in the menu found in the top right corner of the browser; in Chrome, for example, you click on the three-dot menu, then choose "More Tools", then "Developer Tools".  This will open a window at the bottom of the page where you can find information such as errors logged in the console or what network requests were made.


Factors to Consider When Trying to Reproduce a Bug

• User- what user was utilized when the bug was seen?  Did the user have a specific permission level?  You may be dealing with a bug that is only seen by administrators or by a certain type of customer.
• Authentication- was the user logged in?  There may be a bug that only appears when the user is not authenticated, or that only appears when the user is authenticated.
• State of the data- what kind of data does the user have?  Can you try reproducing with exactly the same data?  The bug might only appear when a user has a very long last name, or a particular type of image file.
• Configuration issues- There may be something in the application that isn't set up properly.  For example, a user who isn't getting an email notification might not be getting it because email functionality is turned off for their account.  Check all of the configurable settings in the application and try to reproduce the issue with exactly the same configuration.
• Actions taken before the issue- Sometimes bugs are caused not by the current state where they are seen, but by some event that happened before the current state.  For example, if a user started an action that used a lot of memory, such as downloading a very large file, and then continued on to other activities while the file was downloading, an error caused by running out of memory might affect their current activity.  
• Back button- the Back button can be the culprit in all kinds of mysterious bugs.  If a user navigates to a page through the Back button, the state of the data on the page might be different from what it would be through standard forward navigation.
• Caching- caching can result in unexpected behavior as well. For example, it might appear as if data is unchanged when it in fact has been changed.  If a cache never expires or takes too long to expire, the state of the data can be very different from what is displayed.
• Race conditions- these issues are very difficult to pin down.  Stepping through the code probably won't help, because when the program is run one step at a time the problem doesn't occur.  The best way to determine if there is a race condition is to run your tests a several times and document the inconsistent behavior.  You can also try clicking on buttons or links before a page has loaded in order to speed up input, or throttling your internet connection in order to slow down input.  
• Browser/Browser Version- Browsers are definitely more consistent in their behavior than they used to be, and most browsers are now updated to the latest version automatically, but it's still possible to find bugs that only appear in certain browsers or versions.  If your end user is using IE 8 on an old Windows desktop, for example, it's extremely likely that your application will behave differently for them.
• Browser Size- if a customer is saying that they don't see a Save button or a scrollbar in their browser window, ask them to run a browser size tool in another tab on their browser.  Set your browser to be the same size as theirs, and see if you now have the same problem.
• Machine or Device- Mobile devices are highly variable, so it's possible that a user is seeing an issue on an Android device that you are not seeing on an iOS device, or that a user is seeing a problem on a Motorola device when you are not seeing it on a Samsung.  Laptops and desktop computers are less variable, but it is still possible that there is an issue that a Mac owner is experiencing that you don't see in Windows.  Tools like Browserstack, Perfecto, and Sauce Labs can be helpful for diagnosing an issue on a machine or device that you don't own.  

Once You've Reproduced the Bug

Once you've finally reproduced a tricky bug, you might want to just show it to your developer and be done with it.  But your work is not done!  To make sure that the bug is fixed correctly, you'll want to narrow down the steps to reproduce it as much as you can, to be as precise as possible.  The more precise you can be, the easier it will be for the developer to locate the true cause in the code and fix it.

For example, if you were able to reproduce a bug by using an admin user who navigated to the page with the Back button on a Firefox browser, are you sure that you need all three of those conditions to see the bug?  Do you see the bug when you use an admin user and the Back button in Chrome?  If you do, then you can eliminate the browser from your bug report.  What about if you use a non-admin user?  If you see the bug when you are using a non-admin user, you can take that out of the bug report as well.  Now you have narrowed down the issue to just the Back button, giving the developer a clear indication of where to start with a fix.  

End Users Are Depending on Us

As a software tester, I am often annoyed by bugs and poor user experiences I encounter in my day-to-day living.  I frequently say to myself "Who is testing this website?  What were they thinking?  Does this company even have testers?"  We are the last line of defense to keep our customers from frustration, which is why it is important to chase down those elusive bugs.

What to Test When There's Not Enough Time to Test

In last week's post, I discussed the various things we should remember to test before we consider our testing "done".  This prompted a question from a reader: "How can I test all these things when there is very limited time for testing?"  In today's agile world, we often don't have as much time as we feel we need to fully test our software's features.  Gone are the days when testers had weeks or months to test the upcoming release.  Because software projects usually take longer than estimated, we may be asked to test things at the last minute, just a day or two before the release.  Today I'll discuss what to test when there's not enough time to test, and I'll also suggest some tips to avoid this problem in the first place.

The Bare Minimum: What to Test When There's Almost No Time

Let's use our hypothetical Superball Sorter as an example.  For those who haven't read my series of posts on this feature, the feature takes a number of superballs and sorts them among four children using a set of defined rules. What would I do if I was asked to test this feature for the first time, and it was due to be released tomorrow?

1. Test the most basic case

The first thing I would do would be to test the most basic use case of the feature.  In this case, it would be running the Superball Sorter with no rules at all.  I would test this first because it would give me a very clear indication whether the feature was working at all.  If it wasn't, I could raise the alarm right away, giving the developer more time to fix it.

2. Test the most typical customer scenario

In the case of the Superball Sorter, let's say that we've been told by the product owner that in the most typical scenario, two of the children will be assigned a rule, and the rule will be by size rather than color.  So the next test I would run would be to assign one child a rule that she only accepts large balls, and another child a rule that he only accepts small balls.  I would run the sorter with these rules and make sure that the rules were respected.

3. Run a basic negative test

We all know how frustrating it can be to make a mistake when we try to do an activity online, such as filling out a form, and we have an error on the page but we haven't been given any clear message about what it is.  So the next thing I would test would be to make a common mistake that a user would make and ensure that I got an appropriate error message.  For the Superball Sorter, I would set four rules that resulted in some balls not being able to be sorted, and I would verify that I got an error message that told me this was the case.

4. Test with different users or accounts

Just because something is working correctly for one user or account doesn't mean it's going to work correctly for everyone!  Developers sometimes check their work with only one test user if they are in a big hurry to deliver their feature.  So I would make sure to run the Superball Sorter with at least two users, and I would make sure that those users were different from the one the developer used.

After running these four tests, I would be able to say with some certainty that:

• the feature works at its most basic level
• a typical customer scenario will work correctly
• the customer will be notified if there is an error

If I had time left over after this testing, I would move on to test more Happy Path scenarios, and then to the other tests I mentioned in last week's post.  

Remember that it will never be perfect, and things will never be completely done

When software is released monthly, weekly, or even daily, there's no way to test everything you want to test.  Even if you could get to everything, there will always be some sneaky bug that slips through.  This is just a fact of life in software development.  The good news is that because software is released so frequently, a bug fix can be released very shortly after the bug is found.  So relax, and don't expect things to be perfect.

Speak up- in person and in writing- if disaster is about to strike

Early in my testing career, I was on a team where we were asked to test a large number of new features for a release in a short amount of time.  When we were asked whether we felt confident in the new release, every single one of us said no.  We each delineated the things we hadn't been able to test yet, and why we were concerned about the risks in those areas.  Unfortunately, management went ahead and released the software anyway, because there was a key customer who was waiting for one of the features.  As a result, the release was a failure and had to be recalled after many customer complaints.  

If you believe that your upcoming software release is a huge mistake, speak up!  Outline the things you haven't tested and some of the worst-case scenarios you can envision.  Document what wasn't tested, so that the key decision-makers in your company can see where the risks are.  If something goes wrong after the release, your documentation can serve as evidence that you had concerns.

Enlist the help of developers and others in your testing

While testers possess a valuable set of skills that help them find bugs quickly, remember that all kinds of other people can run through simple test scenarios.  If everyone on your team understands that you have been given too short an amount of time in which to test, they will be happy to help you out.  If I were asking my teammates to test the Superball Sorter, I might ask one person to test scenarios with just one rule, one person to test scenarios with three rules, and one person to test scenarios with four rules, while I continued to test scenarios with two rules.  In this way, we could test four times as many Happy Path scenarios as I could test by myself.  

Talk with your team to find out how you can start testing earlier

To prevent last-minute testing, try to get involved with feature development sooner in the process.  Attend meetings about how the feature will work, and ask questions about integration with other features and possible feature limitations.  Start putting together a test plan before the feature is ready. Work with your developer to write some automated tests that he or she can use while in development.  Ask your developer to commit and push some of their code so you can test basic scenarios, with the understanding that the feature isn't completely done.  In the case of the Superball Sorter, I could ask the dev to push some code once the sorter was capable of sorting without any rules, just to verify that the balls were being passed to each child evenly.  

Automate as much as possible

In sprint-based development, there's often a lull for testers at the beginning of a sprint while the developers are still working on their assigned features.  This is the perfect time to automate features that you have already tested.  When release day looms, much or all of your regression testing can run automatically, freeing you up to do more exploratory testing on the new features.

As testers, we want our users to have a completely bug-free experience.  Because of that, we always want more time for testing than we are given.  With the strategies above, we can ensure that the most important things are tested and that with each sprint we are automating more tests, freeing up our valuable time.  

The One Question to Ask to Improve Your Testing Skills

We've all been in this situation: we've tested something, we think it's working great, and after it goes to Production a customer finds something obvious that we missed.  We can't find all the bugs 100% of the time, but we can increase the number of bugs we find with this one simple question:

"What haven't I tested yet?"  

I have asked this question of myself many times; I make a habit of asking it before I move any feature to Done.  It almost always results in my finding a bug.  The conversation with myself usually goes like this:

Good Tester Me:  "What haven't we tested yet?"  "Well, we haven't tested with an Admin user."
Lazy Tester Me: "Why should that make a difference?  This feature doesn't have anything to do with user privileges."
Good Tester Me: "That may be the case, but we should really test it anyway, to be thorough."
Lazy Tester Me: "But I've been testing this feature ALL DAY!  I want to move on to something else."
Good Tester Me: "You know that we always find the bugs in the last things we think of to test.  TEST IT!"

And I'm always happy I did.  Even if I don't find a bug, I have the peace of mind that I tested everything I could think of, and I've gained valuable product knowledge that I can share with others.

When I ask myself this question, here are twelve follow-up questions I ask:

Did I test with more than one user? 
It seems so obvious, but we are often so embroiled in testing a complicated feature that we don't think to test it with more than our favorite test user.  Even something as simple as the first letter of a last name could be enough to trigger different behavior in a feature.

Did I test with different types of users?
Users often come with different privileges.  When I was first starting out in testing, I would often test with an admin user, because it was the easiest thing to do.  Finding out I'd missed a bug where a regular user didn't have access to a feature they should have taught me a valuable lesson!

Did I test with more than one account/company? 
For those of us testing B2B applications, we often have customers from different accounts or companies. I missed a bug once where the company ID started with a 0, and the new feature hadn't been coded to handle that.

Did I test this on mobile?
Anyone who has ever tested an application on mobile or tablet knows that it can behave very differently from what is seen on a laptop.  You don't want your users to be unable to click a "Submit" button because it's off-screen and can't be accessed.

Did I test this on more than one browser? 
Browsers have more parity in behavior than they did a few years ago, but even so, you will be occasionally surprised by a link that will work in some browsers but not others.

Did I try resizing the browser?
I often forget to do this.  One things I've discovered when resizing is that the scroll bar can disappear, making it impossible for users to scroll through records.

Did I test with the Back button? 
This seems so simple, but a lot of bugs can crop up here!  Also be sure to test the Cancel button on a form.

Is this feature on any other pages, and have we tested on those pages? 
This one recently tripped up my team.  We forgot to test our feature on a new page that's currently in beta.  Be sure to mentally run through all the pages in your application and ask yourself if your feature will be on those pages.  If you have a really large application, you may want to ask testers from other teams in your organization.

Did I test to make sure that this feature works with other features? 
Always think about combining your features.  Will your search feature work with your notification feature?  Will your edit feature work with your sorting feature? And so on.

Have I run negative tests on this feature? 
This is one that's easy to forget when you are testing a complicated feature.  You may be so focused on getting your application configured correctly for testing that you don't think about what happens when bad data is passed in.  For UI tests, be sure to test the limits of every text field, and verify that the user gets appropriate error messages.  For API tests, be sure to pass in invalid data in the test body, and try using bad query parameters.  Verify that you get 400-level responses for invalid requests rather than a generic 500 response.

Have I run security tests on this feature?
It's a sad fact of life that not all of our end users will be legitimate users of our application.  There will be bad actors looking for security flaws to exploit.  This is especially true for financial applications and ones with a lot of personally identifiable information (PII).  Protect your customers by running security scans on your features.

Have I checked the back-end database to make sure that data is being saved as I expected?
When you fill out and submit a form in your application, a success message is not necessarily an indication that the data's been saved.  There could be a bug in your software that causes an error when writing to the database.  Even if the data has been saved, it could have been saved inaccurately, or there may be an error when retrieving the data.  For example, a phone number might be saved with parentheses and dashes, but when the data is retrieved the front-end doesn't know how to parse those symbols, so the phone number isn't displayed.  Always check your back-end data for accuracy.

How is the end user going to use this feature?  Have I run through that scenario?
It's so easy to get wrapped up in our day-to-day tasks of testing, writing automation, and working with our team that we forget about the end user of our application.  You should ALWAYS understand how your user will be using your feature.  Think about what journey they will take.  For example, in an e-commerce app, if you're testing that you can pay with PayPal, make sure you also run through a complete journey where you add a product to your cart, go to the checkout page, and then pay with PayPal.

Missing a bug that then makes it to Production can be humbling!  But it happens to everyone.  The good news is that every time this happens, we learn a new question to ask ourselves before we stop testing, making it more likely that we'll catch that bug next time.

What questions do you ask yourself before you call a feature Done?  Let me know in the comments section!  

Five Strategies for Managing Test Automation Data

Has this ever happened to you?  You arrive at work in the morning to find that many of your nightly automated tests have failed.  Upon investigation, you discover that your test user has been edited or deleted.  Your automation didn't find a bug, and your test isn't flaky; it simply didn't work because the data you were expecting wasn't there.  In this week's post, I'll take a look at five different strategies for managing test data, and when you might use each.

Strategy One: Using data that is already present in the system

This is the easiest strategy- there's nothing to do for setup- but it is also the most risky.  Even if you label your user with "DO NOT REMOVE" there's always a chance that some absent-minded person will delete it.  

However, this strategy can work well if you are just making simple requests.  For example, if you are testing getting a list of contacts, you can assert that contacts were returned.  For the purposes of your test, it doesn't matter what contacts were returned; you just need to know that some contacts were returned.  

Strategy Two: Updating or creating data as a setup step

Most automated test platforms offer the ability to create a setup step that either runs before each test or before a suite of tests.  This strategy works well if it's easy to create or update the record you want to use.  I have a suite of automated API tests that test adding and updating a user's contact information.  Before the tests begin, I run requests that delete the user's email addresses and phone numbers.  

The downside to this strategy is that sometimes my requests to delete the user's contact information fail.  When this happens, my tests fail.  Also, updating data as a setup step adds more time to the test suite, which is something to consider when you need fast results.

Strategy Three:  Using test steps to create and delete data

This is a good strategy when you are testing CRUD (Create, Read, Update, Delete) operations, because you can use the actual tests to create and delete your test data.  If I was testing an API for a contact list, for example, I would have my first test create the contact and assert that the contact was created.  Then I would update the contact and assert that the contact was updated.  Finally, I would delete the contact and assert that the contact was deleted.  There is no impact to the database, because I am both creating and destroying the data.  

However, if one of the tests fails, it's likely the others will as well.  If for some reason the application was unable to create the contact, the second test would fail, because there would be nothing to update.  And the third test would fail because the record would not exist to be deleted.  So even though there was only one bug, you'd have three test failures.

Strategy Four:  Taking a snapshot of the database and restoring it after the tests

This strategy is helpful when your tests are doing a lot of data manipulation.  You take a snapshot of the database as a setup step for the test suite.  Then you can manipulate all the data you want, and as a cleanup step, you restore the database to its original state.  The advantage to this method is that you don't need to write a lot of steps to undo all the changes to your data.  

But this method relies on having the right data there to begin with.  For instance, if you are planning to do a lot of processing on John Smith's records, and someone happened to delete John Smith before you ran your tests, taking a snapshot of the database won't help; John Smith simply won't be there to test on.  It's also possible that taking a snapshot will be time-consuming, depending on the size of your database.

Strategy Five: Creating a mini-database with the data you need for your tests

In this strategy, you spin up your own database with only the data you need for testing, and when your tests have finished, you destroy the database.  If you are using Microsoft technologies, you could do this with their DACPAC functionality; or you are using Docker, you could create your own database as part of your Docker instance.  With this strategy, there is no possibility of your data ever being incorrect, because it is always brand-new and exactly how you configured it.  Also, because your database will be smaller than your real QA environment database, your tests will likely execute more quickly.  

The downside to this strategy is that it requires a lot of preparation.  You may have to do a lot of research on how your data tables relate to each other in order to determine what data you need.  And you'll need to do a fair amount of coding or configuration to set up the creation and destruction steps.  But in a situation where you want to be sure that your data is right for testing, such as when a developer has just committed new code, this solution is ideal.

All of these strategies can be useful, depending on your testing needs.  When you evaluate how accurate you need your data to be, how likely it is that it will be altered by someone else, how quickly you need the tests to run, and how much you can tolerate the occasional failure, it will be clear which strategy to choose.  

What to Put in a Smoke Test

The term "smoke test" is usually used to describe a suite of basic tests that verify that all the major features of an application are working.  Some use the smoke test to determine whether a build is stable and ready for further testing.  I usually use a smoke test as the final check in a deploy to production.  In today's post, I'll share a cautionary tale about what can happen if you don't have a smoke test.  Then I'll continue that tale and talk about how smoke tests can go wrong.

Early in my testing career, I worked for a company that had a large suite of manual regression tests, but no smoke test.  Each software release was difficult, because it was impossible to run all the regression tests in a timely fashion.  With each release, we picked which tests we thought would be most relevant to the software changes and executed those tests.

One day, in between releases, we heard that there had been a customer complaint that our Global Search feature wasn't working.  We investigated and found that the customer was correct.  We investigated further and discovered that the feature hadn't worked in weeks, and none of us had noticed.  This was quite embarrassing for our QA team!

To make sure that this kind of embarrassment never happened again, one of our senior QA engineers created a smoke test to run whenever there was a release to production.  It included all the major features, and could be run fairly quickly.  We felt a lot better about our releases after that.

However, the tester who created the test kept adding test steps to the Smoke Test.  Every time a new feature was created, a step was added to the smoke test.  If we found a new bug in a feature, even it was a small one, a step checking for the bug was added to the smoke test.  As the months went on, the smoke test took longer and longer to execute and became more and more complicated.  Eventually the smoke test itself took so much time that we didn't have time to run our other regression tests.

Clearly there needs to be a happy medium between having no smoke test at all, and having one that takes so long to run that it's no longer a smoke test.  In order to decide what goes in a smoke test, I suggest asking these three questions:

1. What would absolutely embarrass us if it were broken in this application?

Let's use an example of an e-commerce website to consider this question.  For this type of website, it would be embarrassing or even catastrophic if a customer couldn't:

• search for an item they were looking for
• add an item to their cart
• log in to their account
• edit their information

So at the very least, a smoke test for this site should include a test for each of these features.

2. Is this a main feature of the application?

Examples of features in an e-commerce website that would be main features, but less crucial ones, might be:

• wish list functionality
• product reviews
• recommendations for the user

If these features were broken, it wouldn't be catastrophic, but they are features that customers expect.  So a test for each one should be added.

3. If there was a bug here, would it stop the application from functioning?

No one wants to have bugs in their application!  But some bugs are more important than others.  If the e-commerce website had an issue where their "Add to Cart" button was off-center, it might look funny, but it wouldn't stop customers from shopping.  

But a bug where a customer couldn't remove an item from their cart might keep them from checking out with the items they want, which would affect sales.  So a test to check that items can be removed from a cart would be important in a Smoke Test.

With these questions in mind, here is an example of a smoke test that could be created for an e-commerce site:

1. Log in

2. Verify product recommendations are present

3. Do a search for a product

4. Read a review of a product

5. Add an item to the cart

6. Add a second item to the cart and then delete it

7. Edit customer information

8. Check out

9. Write a review

A smoke test like this wouldn't take very long to execute manually, and it would also be easy to automate.  

Whenever new features are added to the application, you should ask yourself the first two questions to determine whether a test for the feature should be added to the smoke test.  And whenever a bug is found in the product, you should ask yourself the third question to determine whether a test for that issue should be added to the smoke test.

Because we want our applications to be of high quality, it's easy to fall into the trap of wanting to test everything, all the time.  But that can create a test burden that keeps us so busy that we don't have time for anything else.  Creating a simple, reliable smoke test can free us up for other activities, such as doing exploratory testing on new features or creating nightly automated tests.