The Positive Outcomes of Negative Testing

As software testers and automation engineers, we often think about the "Happy Path"- the path that the user will most likely take when they are using our application.  When we write our automated UI tests, we want to make sure that we are automating those Happy Paths, and when we write API automation, we want to verify that every endpoint returns a "200 OK" or similar successful response.

But it's important to think about negative testing, in both our manual and automated tests.  Here are a few reasons why:

Our automated tests might be passing for the wrong reasons.

When I first started writing automated UI tests in Javascript, I didn't understand the concept of the promise.  I just assumed that when I made a request to locate an element, it wouldn't return that element until it was actually located.  I was so excited when my tests started coming back with the green "Passed" result, until a co-worker suggested I try to make the test fail by asserting on a different value.  It passed again, because it was actually validating against the promise that existed, which was always returning "True".  That taught me a valuable lesson- never assume that your automated tests are working correctly just because they are passing.  Be sure to run some scenarios where your tests should fail, and make sure that they do so.  This way you can be sure that you are really testing what you think you are testing.

Negative testing can expose improperly handled errors that could impact a user.

In API testing, any client-related error should result in a 400-level response, rather than a 500-level server error.  If you are doing negative testing and you discover that a 403 response is now coming back as a 500, this could mean that the code is no longer handling that use case properly.  A 500 response from the server could keep the user from getting the appropriate information they need for fixing their error, or at worst, it could crash the application.

Negative testing can find security holes.

Just as important as making sure that a user can log in to an application is making sure that a user can't log into an application when they aren't supposed to.  If you only run a login test with a valid username and password, you are missing this crucial area!  I have seen a situation where a user could log in with anything as the password, a situation where a user could log in with a blank password, and a situation where if both the username and password were wrong the user could log in.

It's also crucial to verify that certain users don't have access to parts of an application.  Having a carefully tested and functional Admin page won't mean much if it turns out that any random user can get to it.

Negative testing keeps your database clean.

As I mentioned in my post two weeks ago on input validation, having good, valid data in your database will help keep your application healthy.  Data that doesn't conform to expectations can cause web pages to crash or fail to load, or cause information to be displayed incorrectly.  The more negative testing you can do on your inputs, the more you can ensure that you will only have good data.

For every input field I am responsible for testing, I like to know exactly which characters are allowed.  Then I can run a whole host of negative tests to make sure that entries with the forbidden characters are refused.

Sometimes users take the negative path.

It is so easy, especially with a new feature that is being rushed to meet a deadline, to forget to test those user paths where they will hit the "Cancel" or "Delete" button.  But users do this all the time; just think about times where you have thought about making an online purchase and then changed your mind and removed an item from your cart.  Imagine your frustration if you weren't able to remove something from your cart, or if a "Cancel" button didn't clear out a form to allow you to start again.  User experience in this area is just as crucial as the Happy Path.

Software testing is about looking for unexpected behaviors, so that we find them before a user does.  When negative testing is combined with Happy Path testing, we can ensure that our users will have no unpleasant surprises.