What API Tests to Automate, and When to Automate Them

Last week, we talked about running API tests from the command line using Newman, and how to add Newman into your Continuous Integration system so that your API tests run automatically.  But knowing how to run your tests isn't that helpful unless you make good choices about what tests to run, and when to run them.  Let's first think about what to automate.

Let's imagine that we have an API with these requests:

POST user
GET user/{userId}
PUT user/{userId}
DELETE user/{userId}

The first category of tests we will want to have are the simple Happy Path requests.  For example:

POST a new user and verify that we get a 200 response
GET the user and verify that we get a 200 response, and that the correct user is returned
PUT an update to the user and verify that we get a 200 response
DELETE the user, and verify that we get a 200 response

The next category of tests we want to have are some simple negative requests.  For example:

POST a new user with a missing required field and verify that we get a 400 response
GET a user with an id that doesn't exist and verify that we get a 404 response
PUT an update to the user with an invalid field and verify that we get a 400 response
DELETE a user with an id that doesn't exist and verify that we get a 404 response

You'll want to test 400 and 404 responses on every request that has them.  It's not necessary to test every single trigger of a 400- for example, you don't need to have automated tests for every single missing required field- but you will want to have one test where one required field is missing.

The third category of tests we want to have are more Happy Path requests, with variations. For example:

POST a new user with only the required fields, rather than all fields, and verify that we get a 200 response
GET a user with query parameters, such as user/{userId}?fields=firstName,lastName, and verify that we get a 200 response, and the appropriate values in the response
PUT a user where one non-required field is replaced with null, and one field that is currently null is replaced with a value, and verify that we get a 200 response

It's worth noting that we might not want to test every possible combination in this category.  For example, if our GET request allows us to filter by five different values: firstName, lastName, username, email, and city, there are dozens of possibilities of what you could filter on.  We don't want to automate every single combination; just enough to show that each filter is working correctly, and that some combinations are working as well.

Finally, we have the category of security tests.  For example, if each request needs an authorization token to run, verify that we get an appropriate error:

POST a new user without an authorization token, and verify that we get a 401 response
GET a user with an invalid token, and verify that we get a 403 response
PUT an update to the user with an invalid token, and verify that we get a 403 response
DELETE a user without an authorization token, and verify that we get a 401 response

For each request, you'll want to test for both a 401 response (an unauthenticated user's request) and a 403 response (an authorized user's request).

There may be many more tests than the ones that have been listed here that are appropriate for an API you are testing.  But these examples serve to get you thinking about the four different types of tests.

Now let's take a look at how we might use these four types in automation!  First, we want to have some Smoke Tests that will run very quickly when code is deployed from one environment to another, up the chain to the Production environment.  What we want to do with these tests is simply verify that our endpoints can be reached.  So all we need to do is run the first category of tests: the simple Happy Path requests.  In our example API, we only have four request types, so we only need to run four tests.  This will only take a matter of seconds.

We'd also like to have some tests that run whenever new code is checked in.  We want to make sure that the new code doesn't break any existing functionality.  For this scenario, I recommend doing the first two categories of tests: the simple Happy Path requests, and the simple negative requests.  We could have one positive and one or two negative tests for each request, and this will probably be enough to provide accurate feedback to developers when they are checking in their code.  In our example API, this amounts to no more than twelve tests, so our developers will be able to get feedback in about one minute.

Finally, it's also great to have a full regression suite that runs nightly.  This suite can take a little longer, because no one is waiting for it to run.  I like to include tests of all four types in the suite, or sometimes I create two nightly regression suites: one that has the first three types of tests, and one that has just the security tests.  Even if you have a hundred tests, you can probably run your full regression suite in just a few minutes, because API tests run so quickly.

Once you have your Smoke, Build, and Regression tests created and set to run automatically, you can relax in the knowledge that if something goes wrong with your API, you'll know it.  This will free you up to do more exploratory testing!

Next week we'll take look at API request formats: JSON structure and query parameters!