Rethinking the Pyramid: The Automation Test Wheel

Anyone who has spent time working on test automation has likely heard of the Test Automation Pyramid.  The pyramid is typically made of three horizontal sections: UI Tests, API Tests, and Unit Tests.  The bottom section is the widest section, and is for the unit tests.  The idea is that there should be more unit tests run than any other kind of tests.  The middle section is for the API tests, and the idea is that fewer API tests should be run than unit tests.  Finally, the top section is for the UI tests, and the idea is that the least number of tests run should be UI tests, because they take the most time and are the most fragile.

In my opinion there are two things wrong with the pyramid: it leaves out many types of automated tests, and it assumes that the number of tests is the best indicator of appropriate test coverage.  I propose a new way of thinking about automated testing:  the Automation Test Wheel.

Each of these test types can be considered as spokes in a wheel; none is more important than another, and they are all necessary.  The size of each section of the wheel does not indicate the quantity of the tests to be automated; each test type should have the number of tests that are needed in order to verify quality in that area.  Let's take a look at each test type.

Unit Tests:  A unit test is the smallest automated test possible.  It tests the behavior of just one function or method.  For example, if I had a method that tested whether a number was zero, I could write these unit tests:

• A test that passes a zero to the method and validates that it is identified as a zero
• A test that passes a one to the method and validates that it is identified as non-zero
• A test that passes a string to the method and validates that the appropriate exception is thrown

Because unit tests are independent of all other services and because they run so quickly, they are a very effective way of testing code.  They are often written by the developer who wrote the method or function, but they can also be written by others.  Each method or function should have at least one unit test associated with it.

Component Tests: These tests check the various services that the code is dependent on.  For example, if we had code that called the GitHub API, we could write a component test that would make a call to the API and verify that the API was running.  Other examples of component tests are pinging a server or making a call to a database and verifying that a response was received.  There should be at least one component test for each service the code relies on.

Services Tests:  These are tests that check the web services that are used in our code.  In today's applications, web services often use API requests.  For example, if we have an API with POST, GET, PUT, and DELETE requests, we will want to have automated tests that check each request type.  We will want to have both "happy path" tests that check that a valid request returns an appropriate response, and also negative tests that verify that an invalid request returns an appropriate error code.

User Interface (UI) Tests: UI tests verify that end-user activities work correctly.  These are the tests that will fill out text fields and click on buttons.  As a general rule, anything that can be tested with a unit, component, or service test should be tested by those methods instead.  UI tests should focus solely on the user interface.

Visual Tests: Visual tests verify that elements are actually appearing on the screen.  This is slightly different from UI tests, because the UI tests are focusing on the functionality of the user interface rather than the appearance.  Examples of visual tests would be: verifying that a button's label is rendered correctly, and verifying that the correct product image is appearing on the screen.

Security Tests:  These are tests that verify that security rules are being respected.  These tests can overlap with services tests, but should still be considered separately.  For example, a security test could check to make sure that an authorization token cannot be generated with an invalid username and password combination.  Another security test would be to make a GET request with an authorization token for a user who should not have access to that resource, and verify that a 403 response is returned.

Performance Tests: Automated performance tests can verify that request response times happen within an appropriate time period.  For example, if your company has decided that GET requests should never take longer than two seconds, test requests can be set to return a failure state if the request takes longer than that time.  Web page load times can also be measured with performance tests.

Accessibility Tests:  Automated accessibility tests can check a variety of things.  When combined with UI tests, they can verify that images have text descriptions for the visually impaired.  Visual tests can be used to verify that the text on the screen is the correct size. 

You may have noticed that the above descriptions often overlap each other.  For example, security tests might be run through API testing, and visual tests might be run through UI testing.  What is important here is that each area is tested thoroughly, efficiently, and accurately.  If there is a spoke missing from the wheel, you will never be comfortable relying on your automation when you are doing continuous deployment. 

Next week, I'll discuss how we can fit all these tests into a real-world application testing scenario!

Fifteen Free Tools to Help With Testing

There are a great many articles, blog posts, and presentations that discuss automation frameworks and strategies.  But even the most robust automation framework won't eliminate the need to do exploratory testing.  There will always be situations where we need to generate a large amount of text to test a text field or where we need to encode a string in HTML to test for cross-site scripting.  In this week's post, I share fifteen of my favorite free tools that make testing faster and easier.  

Text Tools:

1. Letter Count:  This tool will count the characters or words in a block of text.  I use it for creating strings with a specific character count when I test text fields.

2. Lorem Ipsum Generator: I use this tool when I need to generate large amounts of text for text fields where a user will be able to enter several paragraphs of text.

3. Convert Case: This tool comes in handy when I'm testing with Postman and my assertions are expecting the exact casing for string comparison.  Convert Case will set all the characters in a string to lower case, upper case, sentence case, alternating case, and more.

JSON Tools:

4. Pretty Print: JSON objects need indentation to be easily readable.  This tool will take care of all of the indentation and spacing for you.  This is especially helpful when you receive flattened JSON in a response and you want to be able to read through it.

5. Online JSON Viewer: This tool will flatten your JSON for you by removing all the white spaces, which is helpful when you need to use your JSON in code.

6. JSON Lint: I use this tool whenever I'm using JSON in a test and getting 400 errors.  I paste the JSON into the tool and it will tell me whether my JSON is valid.  If it's not valid, it shows me the line with the error.

Encoding and Decoding Tools:

7. URL Encoder/Decoder: This tool is great for testing cross-site scripting.  Simply paste in your script and it will encode it in valid HTML format for you.

8. JWT.IO: This official JWT documentation includes a tool that will decode a JWT.  It's really helpful for testing authorization, because you can see exactly what information your JWT is sending.  

9. Base 64 Decode and Encode: When you have an image or string that's been encoded into Base 64, this is an easy way to decode it.  

GUID Tools:

10. GUID Generator: I often need random GUIDs for my testing, and this tool will generate as many as I need.

11. GUID Test and Validate: I use this tool to check GUIDs to make sure that they are valid. 

Miscellaneous Tools:

12. Rubular: This is a regex editor that allows you to put in a regex, add a string to test, and find out whether your string matches the regex.  It is specifically designed for Ruby but can be used with other languages as well.  

13. NumVerify: This tool is helpful when you are testing international phone numbers.  It lets you know whether a given number is valid, and which country it is valid in.  

14. WePay Testing: I use this site when I need a fake bank account to test with.  It also lists fake credit card numbers for use in testing.  Some of the features may be specific to Chase Bank.  

15. Online Random File Generator: When you need to test files of a very specific size, this tool uses random strings to generate a file with exactly the size you need.

Just as we use tools in our daily life to make our tasks easier, these free online tools can help speed up our testing, giving us more time to write test automation or do further exploratory testing.  

What are your favorite free tools for testing?  Let me know in the comments below!  

Six Steps to Writing an Effective Test Report

As testers, we know how important it is to test our software thoroughly and document our findings meticulously.  But all of our talent will be useless if we can't effectively communicate our test results to others!  If your test results are written in a giant, poorly organized spreadsheet with tiny text and lots of unnecessary details, even the most dedicated test manager will find her eyes glazing over with boredom when she looks at it.  In this post, I'll describe six steps to take to make sure that you can communicate your findings to others efficiently and effectively.

Step One: Determine what goal you are trying to accomplish with your report

Why are you creating your report?  What do you want people to understand when they read it?  You might be creating a report to demonstrate that you have tested all the acceptance criteria in the user story.  You could be showing your manager exactly which areas of the application are not functioning properly.  You could be demonstrating that after a recent code change several bugs were fixed.  

There are all kinds of reasons to create a report, but if you don't stop and think about why you are creating it, it's probably not going to be very clear to your readers.  Simply sending over your test notes is not enough to communicate effectively.

Step Two: Focus on the viewer's needs

Who will be reading your report?  Will it be a developer, your test manager, your product owner, your team lead, or the CTO of your company?  You will want to tailor your test report for your audience.  The CTO of your company is probably very busy and will not care how many permutations of your form data you ran, or how you developed the permutations.  She will want to see that you ran 100 tests and that 99 passed.  Your product owner will want to see that you have tested the user stories and that the outcomes were as expected.  Your test manager might be interested in how many different permutations of the test you ran, and what logic you used to create them.  Your developer might want to know what test data you used and what your server response times were.  

You can see how the interests of your reader will vary quite a bit depending on their role, so think about how you can best present the information that he or she wants.

For steps three through six, we'll once again use the example of the Superball Sorter.  This is a hypothetical software feature that sorts out superballs of varying sizes and colors to four children according to a set of rules.  

Step Three: Avoid Extraneous Information

Make sure that your report contains only the information that your reader needs.  Extraneous information means that a reader has to sift through your report to find the important results.  Consider this report, which shows the results of two tests where two children have a sorting rule:

Rules	Amy	Bob	Carol	Doug
Amy-small blue balls; Doug- large green balls	small blue ball; small blue ball; small blue ball; small blue ball	large red ball; small orange ball; large yellow ball; small purple ball	large purple ball; small green ball; large yellow ball; small red ball	large green ball; large green ball; large green ball; large green ball
Bob- large red balls; Carol- small yellow balls	large orange ball; small purple ball; large yellow ball; small green ball	large red ball; large red ball; large red ball; large red ball; large red ball	small yellow ball; small yellow ball; small yellow ball; small yellow ball	small blue ball; small green ball; large purple ball; small orange ball

Did the tests pass?  How long did it take for you to determine that by looking at the test report?  There is a lot of information here that is unnecessary.  It doesn't matter that Bob got the large red ball first and the small orange ball second.  What matters is that Amy only got small blue balls and Doug got large green balls.  Contrast that test report with this one:

Rules	Rules respected?
Amy-small blue balls; Doug- large green balls	Yes
Bob- large red balls; Carol- small yellow balls	Yes

Here you can very quickly see what rules were set, and whether those rules were respected.  If a reader of your report needs to know what balls Bob got, they can ask you for those details and you can look them up in your notes.

Step Four: Make the report visually immediate

We are all busy people; developing and testing software is fast-paced and time-consuming!  Your manager or CTO is probably sent dozens of reports and emails a day, so make your report so easy to read that just a glance at it will give them information.  Take a look at this test report:

Number of Rules	Pass/Fail
0
1
2
3
4

How many seconds does it take you to see that a test failed?  This report is pretty immediate!  It's also really easy to see how many rules were used when the test failed.  Compare that with this test report, which shows the same tests and the same results:

Test Case	Result
None of the children have rules	The balls are sorted evenly amongst the children
One child has a rule	The child’s rule is respected
Two children have rules	The two children’s rules are respected
Three children have rules	The three children’s rules are respected
Four children have rules	None of the balls are sorted

A reader needs to read through all the text of this report in order to see that the fourth test failed.

Step Five: Make the report easy to read

In addition to making the report visually immediate, it also needs to be easy to read. Take a look at this example, where two tests are run where three children are given rules:

Rules	Rules respected?
Amy- small blue; Bob- large blue; Carol- small purple	Amy gets only small blue balls, and Bob gets only large blue balls, but Carol gets balls other than the small purple balls
Amy- large blue; Bob- small purple; Carol- small yellow	Amy gets only large blue balls, Bob gets only small purple balls, and Carol gets only small yellow balls

This report shows very quickly that one of the tests failed, but in order to see why the test failed, it's necessary to read through the whole description to see that Carol's rule was not respected.  This report conveys exactly the same information:

Rules	Amy	Bob	Carol
Amy- small blue; Bob- large blue; Carol- small purple	PASS	PASS	FAIL
Amy- large blue; Bob- small purple; Carol- small yellow	PASS	PASS	PASS

With this report, it is so easy to see that Carol's rule was not respected, and it's also easy to look to the left to see which rule she had.

Step Six: Make the report readable without any additional explanation

How long does it take for you to figure out what this report means? 

Rules	Result
A-SB; B-LO; C-L; D-S	A-Y; B-Y; C-Y; D-N
A-L; B-S; C-Y; D-P	A-Y; B-N; C-Y; D-Y
A-LY; B-L; C-S; D-SG	A-Y; B-Y; C-N; D-Y

It's fine to use all sorts of abbreviations when you are testing and taking notes for yourself, but you shouldn't need to provide a key to your reader in order to have them interpret it.  Who but the most interested testers are going to take the time to see where the bug is here?

This report conveys exactly the same information:

Test One	Amy- small blue	Bob- large orange	Carol- large	Doug- small
Rule respected?	Yes	Yes	Yes	No
Test Two	Amy- large	Bob- small	Carol- yellow	Doug- purple
Rule respected?	Yes	No	Yes	Yes
Test Three	Amy- large yellow	Bob- large	Carol- small	Doug- small green
Rule respected?	Yes	Yes	No	Yes

It's easy to see exactly what rules each child was given for each test.  Through the use of color, the report demonstrates very clearly where the bug is: whenever a child is given a rule that they should get only small balls, that rule is not respected.

Conclusion:

In today's fast-paced world, we all have vast amounts of information coming to us every day.  If we are going to make a difference with our testing and influence decision-making where we work, we need to be able to convey our test results in ways that clearly show what is going on with our software and what should be done to improve it.

How to Reproduce a Bug

Have you ever seen something wrong in your application, but you haven't been able to reproduce it?  Has a customer ever reported a bug with a scenario that you just couldn't recreate?  It is tempting to just forget about these bugs, but chances are if one person has seen the issue, other people will see it as well.  In this post I'll discuss some helpful hints for reproducing bugs and getting to the root cause of issues.

Gather Information

The first thing to do when you have a bug to reproduce is to gather as much information as you can about the circumstances of the issue.  If it's a bug that you just noticed, think about the steps that you took before the bug appeared.  If it's a bug that someone else has reported, find out what they remember about the steps they took, and ask for details such as their operating system, browser, and browser version.

One Step at a Time

Next, see if you can follow the steps that you or the reporter of the bug took.  If you can reproduce the bug right away, you're in luck!  If not, try making one change at a time to your steps, and see if the bug will appear.  Don't just thrash around trying to reproduce the issue quickly; if you're making lots of disorganized changes, you might reproduce the bug and not know how you did it.  Keep track of the changes you made so you know what you've tried and what you haven't tried yet.

Logs and Developer Tools

Application logs and browser developer tools can be helpful in providing clues to what is going on behind the scenes in an application.  A browser's developer tools can generally be accessed in the menu found in the top right corner of the browser; in Chrome, for example, you click on the three-dot menu, then choose "More Tools", then "Developer Tools".  This will open a window at the bottom of the page where you can find information such as errors logged in the console or what network requests were made.


Factors to Consider When Trying to Reproduce a Bug

• User- what user was utilized when the bug was seen?  Did the user have a specific permission level?  You may be dealing with a bug that is only seen by administrators or by a certain type of customer.
• Authentication- was the user logged in?  There may be a bug that only appears when the user is not authenticated, or that only appears when the user is authenticated.
• State of the data- what kind of data does the user have?  Can you try reproducing with exactly the same data?  The bug might only appear when a user has a very long last name, or a particular type of image file.
• Configuration issues- There may be something in the application that isn't set up properly.  For example, a user who isn't getting an email notification might not be getting it because email functionality is turned off for their account.  Check all of the configurable settings in the application and try to reproduce the issue with exactly the same configuration.
• Actions taken before the issue- Sometimes bugs are caused not by the current state where they are seen, but by some event that happened before the current state.  For example, if a user started an action that used a lot of memory, such as downloading a very large file, and then continued on to other activities while the file was downloading, an error caused by running out of memory might affect their current activity.  
• Back button- the Back button can be the culprit in all kinds of mysterious bugs.  If a user navigates to a page through the Back button, the state of the data on the page might be different from what it would be through standard forward navigation.
• Caching- caching can result in unexpected behavior as well. For example, it might appear as if data is unchanged when it in fact has been changed.  If a cache never expires or takes too long to expire, the state of the data can be very different from what is displayed.
• Race conditions- these issues are very difficult to pin down.  Stepping through the code probably won't help, because when the program is run one step at a time the problem doesn't occur.  The best way to determine if there is a race condition is to run your tests a several times and document the inconsistent behavior.  You can also try clicking on buttons or links before a page has loaded in order to speed up input, or throttling your internet connection in order to slow down input.  
• Browser/Browser Version- Browsers are definitely more consistent in their behavior than they used to be, and most browsers are now updated to the latest version automatically, but it's still possible to find bugs that only appear in certain browsers or versions.  If your end user is using IE 8 on an old Windows desktop, for example, it's extremely likely that your application will behave differently for them.
• Browser Size- if a customer is saying that they don't see a Save button or a scrollbar in their browser window, ask them to run a browser size tool in another tab on their browser.  Set your browser to be the same size as theirs, and see if you now have the same problem.
• Machine or Device- Mobile devices are highly variable, so it's possible that a user is seeing an issue on an Android device that you are not seeing on an iOS device, or that a user is seeing a problem on a Motorola device when you are not seeing it on a Samsung.  Laptops and desktop computers are less variable, but it is still possible that there is an issue that a Mac owner is experiencing that you don't see in Windows.  Tools like Browserstack, Perfecto, and Sauce Labs can be helpful for diagnosing an issue on a machine or device that you don't own.  

Once You've Reproduced the Bug

Once you've finally reproduced a tricky bug, you might want to just show it to your developer and be done with it.  But your work is not done!  To make sure that the bug is fixed correctly, you'll want to narrow down the steps to reproduce it as much as you can, to be as precise as possible.  The more precise you can be, the easier it will be for the developer to locate the true cause in the code and fix it.

For example, if you were able to reproduce a bug by using an admin user who navigated to the page with the Back button on a Firefox browser, are you sure that you need all three of those conditions to see the bug?  Do you see the bug when you use an admin user and the Back button in Chrome?  If you do, then you can eliminate the browser from your bug report.  What about if you use a non-admin user?  If you see the bug when you are using a non-admin user, you can take that out of the bug report as well.  Now you have narrowed down the issue to just the Back button, giving the developer a clear indication of where to start with a fix.  

End Users Are Depending on Us

As a software tester, I am often annoyed by bugs and poor user experiences I encounter in my day-to-day living.  I frequently say to myself "Who is testing this website?  What were they thinking?  Does this company even have testers?"  We are the last line of defense to keep our customers from frustration, which is why it is important to chase down those elusive bugs.