One Request, Sixteen Assertions

Anyone who has been reading my blog for a while knows that I'm passionate about API testing.  I'm also passionate about using Postman for API testing, because in my opinion it's the easiest way to test API requests.  So it makes me sad when I see people testing an API and only asserting that they got a 200 response!  In this week's post, I'm going to take a simple GET request and show examples of 16 assertions that could be run on the request.

For the GET request, I'll be using the wonderful Restful-Booker application created by Mark Winteringham to teach people how to do API testing.  The request URL I'll be using is https://restful-booker.herokuapp.com/booking/1 which will retrieve the booking with the ID of 1. If you run this GET request in Postman, you'll get a response like this:

The response has the first and last name of a hotel guest, the total price of their room, whether or not their deposit has been paid, and what their checkin and checkout dates are.

You may see different details in the response from those in the above image, because this is a site that is under frequent use as people practice making API requests to change booking information.

Let's look at all the different assertions we can do on this response. To add these assertions yourself, click on the "Tests" tab just underneath the request URL.

1. First, we can add the most common assertion: asserting that the response code is correct.

pm.test("Status code is 200", function () { pm.response.to.have.status(200); });

The pm.test refers to "Postman test". "Status code is 200" is the name of our assertion. And pm.response.to.have.status(200) is what we are expecting from the response.

2. We can also assert that the response comes back within a reasonable time frame, like this:

pm.test("Response time is less than 1000ms", function () { pm.expect(pm.response.responseTime).to.be.below(1000); });

This asserts that the response time is less than one second.

3-8. Next we can assert that expected headers are present:

pm.test("Server header is present", function () { pm.response.to.have.header("Server"); }); pm.test("Connection header is present", function () { pm.response.to.have.header("Connection"); });

pm.test("Content-Length header is present", function () { pm.response.to.have.header("Content-Length"); }); pm.test("Etag header is present", function () { pm.response.to.have.header("Etag"); }); pm.test("Date header is present", function () { pm.response.to.have.header("Date"); }); pm.test("Via header is present", function () { pm.response.to.have.header("Via"); });

These assertions are looking for the presence of a particular header rather than checking that the header has a specific value. The specific assertions above might not be particularly necessary, but sometimes it's a good idea to assert that certain headers are in place, such as X-XSS-Protection (not in this API), which indicates that there are measures in place to protect from cross-site scripting.  

9-10.  We can also have assertions that validate that a specific header value is being returned:

pm.test("X-Powered-By header value is Express", function () {
    pm.response.to.be.header("X-Powered-By", "Express");
});

pm.test("Content-Type header value is application/json", function () {
    pm.response.to.be.header("Content-Type", "application/json; charset=utf-8");
});

The first assertion is validating that the server used is Express, and the second assertion is validating that the Content-Type of the response is application/json.

11.  We can assert that certain text is included in the body of the response:

pm.test("Response contains last name", function () {
    pm.expect(pm.response.text()).to.include("lastname");
});

In this assertion, I am validating that the text in the body of the response contains "lastname".

12-13.  I can also assert that specific json fields exist in the response.  To do this, I first need to parse out the json in the response and save it to a variable:

var jsonData = pm.response.json();

Then I can assert on specific fields in the jsonData variable.  Here I am checking to make sure that the response has both a checkin field and a checkout field:

pm.test("Response contains checkin date", function () {
    pm.expect(jsonData.bookingdates.checkin).to.exist;
})

pm.test("Response contains checkout date", function () {
    pm.expect(jsonData.bookingdates.checkout).to.exist;

})

14-15. I can also assert that specific values are returned in the response:

var jsonData = pm.response.json();

pm.test("Correct first name is returned in response", function () { pm.expect(jsonData.firstname).to.eql("Mark"); });

pm.test("Deposit has been paid", function () { pm.expect(jsonData.depositpaid).to.be.true; });

In this first assertion, I am validating that the value of the first name returned in the response is "Mark". In the second assertion, I'm validating that "depositpaid" is true. Keep in mind that these values may actually change because others are using this API. When you add specific value assertions, you'll want to be sure that your test data does not change; otherwise your tests will be flaky!

16. Finally, I can assert that a value is greater or less than a certain number:

var jsonData = pm.response.json();

pm.test("Total price is greater than 100", function () { pm.expect(jsonData.totalprice).to.be.above(100); });

In this case, I am asserting that the total price paid is greater than 100.

The types of assertions that you run on your API responses will vary depending on what is important for you to test. While you may not decide that all types are appropriate for your API, I hope that these examples show that there are many more things to assert on than simply "200 OK"!

Stop Writing So Many UI Tests!

If you were to guess the importance of various types of automated tests by looking at the number of tutorials and articles about them on the Web, you'd think that UI tests were the most important.  But this is not the case- so much of an application can be tested through other means, especially API tests.  API tests are faster and much less flaky than UI tests, and they're easier to write as well!  Below are four types of tests that are better suited for API testing.

Login Tests:  It's easy to cycle through all kinds of username and password combinations with API tests.  The response time from a POST to a login endpoint is lightning fast, as opposed to a UI test which has to wait for the username and password fields to be populated and wait for the login response to reach the browser.  To prove this, I created a Postman collection that had sixteen login tests with various user and password combinations.  The sixteen tests ran in less than three seconds!  Imagine how long the equivalent UI tests would take.  

However, you should have two automated UI tests: one that validates that the login page looks correct, and that the user is able to log in, and one that validates that an appropriate error message is displayed when the user attempts to log in with bad credentials.

CRUD Tests:  When you're testing CRUD functionality, you're testing how your application interacts with the underlying data store.  This is best done at the API level.  It's easy to create GET, POST, PUT, and DELETE tests using a tool like Postman.  You can assert on both the response codes and the body of the response (if any), and you can also do GETs to assert that your POSTs, PUTs, and DELETEs have saved correctly to the database.

The only UI tests you need in this area are one that demonstrates that form fields can be filled out and submitted, and one that shows that data is displayed correctly on the page.

Negative Tests:  API testing is great for negative tests, because not only can you run through all kinds of negative scenarios very quickly, but you can also run tests that aren't possible in the UI.  For example, let's say that your form has a required field.  In the UI, you can't do a test where you submit a new record without that required field, because the UI simply won't let you.  But in the API, you can do a POST without the required field and verify that you are getting a 400-level response.  API testing is also great for checking application security, because malicious users are likely to try to attack the application at this level.  

Here is just a sampling of the types of negative tests you can run with API testing:

• sending in an inaccurate URL
• trying a request without appropriate authentication
• testing for IDOR
• sending in incorrect headers
• sending in a request without a required field
• trying a request with a field value that violates type or length constraints
• verifying that the correct 400-level error is displayed when a request is invalid

For UI testing, you'll simply want to verify that appropriate errors are displayed on the page when you leave a required field blank or violate a field constraint.  Everything else can be covered by API tests.

Tests of Complicated Business Logic:  If you have an area of your application that requires an extensive setup of data and complicated business logic, it's much easier to test with an API than with the UI.  Consider my hypothetical Superball Sorter, which sorts balls of various colors and sizes among four children.  Setting up the rules through the UI in an automated test would be tedious; assuming each child had a dropdown picker for size and color, you'd need to do a lot of element selection.  But if the Superball Sorter had an API that could set all the rules for the children in a single POST, it would take milliseconds to prepare the test.  

Similarly, after the sorting has been run, a UI test would need to grab all the responses on the page to validate that the balls have been sorted correctly, where an API could do a GET request for each child and validate that the appropriate balls are returned.  Four GET requests will most likely be returned and validated before a UI test could validate a single child's values.  

Now that you have seen the many ways that API tests can be used, I hope that you will take the time to look at your current UI test suite to see which tests could be shifted to API testing.  Your automation suite will be faster, more reliable, and easier to maintain as a result!

The Easiest MongoDB Tutorial on the Web

MongoDB is one of the most popular non-relational databases in use today.  Its versatility, speed, and scalability make it popular with applications that need to store data in a JSON-like format.  It's very easy to install MongoDB and create a database, but the query language it uses is quite different from the SQL query language.  When I first started using MongoDB, I was frustrated by the documentation I found on queries; either they tried to explain too much or the examples were too complicated.  More than once I said things in frustration like "How can I simply ask for 'select lastName from contacts where id = 3?!!'".

It is because of this frustration that I have created this tutorial.  In the tutorial, I'll be including several really simple examples of queries that you are probably used to running in SQL. 

Installing Mongo:

Because this post is really about writing queries, I'm going to skip the installation instructions, and instead send you here for MacOSX and here for Windows.  Once you have finished the installation instructions, you should have a command window open that is running mongo.

Creating a Database:
Creating a new database is unbelievably easy.  We're going to name our new database tutorial.  To create it, simply type use tutorial.  You'll get the response switched to db tutorial.  Tada!  Your database is created.

Adding a Document:
Of course, right now your database is completely empty.  Let's change that by typing
db.tutorial.insertOne( { _id: 1, firstName: "Prunella", lastName: "Prunewhip" } ).  
You will get a response of 
{ "acknowledged" : true, "insertedId" : 1 }. 
You have just added your first document!  Note that a "document" is the equivalent of a "record" in a SQL database.  Your document has an id (which is preceded by an underscore, by convention), a first name, and a last name.

Retrieving All Documents:
Let's make sure that your document was really added by asking for it.  Type 
db.tutorial.find() 
and you should get this as a result: 
{ "_id" : 1, "firstName" : "Prunella", "lastName" : "Prunewhip" }. 
The empty find() command will find all of the documents in the database.  At the moment, we only have one document, so that's all that was returned.

Adding Multiple Documents:
To add several documents at the same time, use the InsertMany command, like this:

db.tutorial.insertMany([ { _id: 2, firstName: "Joe", lastName: "Schmoe" }, { _id: 3, firstName: "Amy", lastName: "Smith" }, { _id: 4, firstName: "John", lastName: "Smith" }, { _id: 5, firstName: "Joe", lastName: "Bagadonuts" }, { _id: 6, firstName: "Carol", lastName: "Jones" }, { _id: 7, firstName: "Robert", lastName: "Johnson" } ]). 

Note that each document is wrapped in curly braces, separated by commas.  You'll get a response like this: 
{ "acknowledged" : true, "insertedIds" : [ 2, 3, 4, 5, 6, 7 ] }. 
Now you have seven records in your database.

If you retrieve all documents at this point using db.tutorial.find(), you'll get a result like this:
{ "_id" : 1, "firstName" : "Prunella", "lastName" : "Prunewhip" }
{ "_id" : 2, "firstName" : "Joe", "lastName" : "Schmoe" }
{ "_id" : 3, "firstName" : "Amy", "lastName" : "Smith" }
{ "_id" : 4, "firstName" : "John", "lastName" : "Smith" }
{ "_id" : 5, "firstName" : "Joe", "lastName" : "Bagadonuts" }
{ "_id" : 6, "firstName" : "Carol", "lastName" : "Jones" }
{ "_id" : 7, "firstName" : "Robert", "lastName" : "Johnson" }

Retrieving a Single Document:
To retrieve a single document, use this syntax:
db.tutorial.find( { _id: 1 } ).  
This will return the document with the id of 1: 
{ "_id" : 1, "firstName" : "Prunella", "lastName" : "Prunewhip" }

Search for All Documents With a Single Value:
The previous search on id will always return just one document, because the id is unique.  If you want to search for all documents that have a particular value, you can use 
db.tutorial.find({ lastName: "Smith"}).  
This will return all documents that have the last name Smith:
{ "_id" : 3, "firstName" : "Amy", "lastName" : "Smith" }
{ "_id" : 4, "firstName" : "John", "lastName" : "Smith" }

Search for One Value in One Document:
Let's say you want to find the last name of the document with the id of 3.  To do this, type:
db.tutorial.find({ _id: 3}, {lastName:1, _id:0}).  
You will get this result:  
{ "lastName" : "Smith" }. 
The _id:0 is there to specify that you don't want the id returned in the response; returning the id in the response is a default behavior in MongoDB.

Return All the Values for a Specific Field:
If you wanted to get a list of all the last names in your database, you would use
db.tutorial.find({}, {lastName:1, _id:0}).  
This would return
{ "lastName" : "Prunewhip" }
{ "lastName" : "Schmoe" }
{ "lastName" : "Smith" }
{ "lastName" : "Smith" }
{ "lastName" : "Bagadonuts" }
{ "lastName" : "Jones" }
{ "lastName" : "Johnson" }

Search with "Starts With":
MongoDB uses regex to search on field values.  To search for all the documents that have last names that begin with S, you'd do this search:
db.tutorial.find({ lastName: /^S/}).  
This will return 
{ "_id" : 2, "firstName" : "Joe", "lastName" : "Schmoe" }
{ "_id" : 3, "firstName" : "Amy", "lastName" : "Smith" }
{ "_id" : 4, "firstName" : "John", "lastName" : "Smith" }

Search with "And":
If you wanted to search for a document that had a specific first name AND a specific last name, you'd search like this:
db.tutorial.find( {$and: [{ lastName: "Smith" },{ firstName: "Amy"}]} )
which would return 
{ "_id" : 3, "firstName" : "Amy", "lastName" : "Smith" }.

Search with "In":
To search for all the documents that have either the last name Smith or the last name Jones, you'd use:
db.tutorial.find({lastName :{$in :["Smith","Jones"]}}).  
This will return
{ "_id" : 3, "firstName" : "Amy", "lastName" : "Smith" }
{ "_id" : 4, "firstName" : "John", "lastName" : "Smith" }
{ "_id" : 6, "firstName" : "Carol", "lastName" : "Jones" }

Update a Document:
If you'd like to change an existing document, you can use the Update command.  For example, to change the last name of the third document from Smith to Jones, you would type:
db.tutorial.updateOne({_id: 3 }, {$set: {lastName: "Jones"}}).  
You'll get this response: 
{ "acknowledged" : true, "matchedCount" : 1, "modifiedCount" : 1 }.

To verify that the record was updated correctly, you can use db.tutorial.find( { _id: 3 } ), which will return { "_id" : 3, "firstName" : "Amy", "lastName" : "Jones" }.

Delete a Document:
Finally, there may be times where you want to delete a document.  This can be done with
db.tutorial.deleteOne({_id: 4 })
which will return a response of 
{ "acknowledged" : true, "deletedCount" : 1 }.

To verify that the document has been deleted, you can run db.tutorial.find() and get this response:
{ "_id" : 1, "firstName" : "Prunella", "lastName" : "Prunewhip" }
{ "_id" : 2, "firstName" : "Joe", "lastName" : "Schmoe" }
{ "_id" : 3, "firstName" : "Amy", "lastName" : "Jones" }
{ "_id" : 5, "firstName" : "Joe", "lastName" : "Bagadonuts" }
{ "_id" : 6, "firstName" : "Carol", "lastName" : "Jones" }
{ "_id" : 7, "firstName" : "Robert", "lastName" : "Johnson" }
and you can see that the document with the id of 4 is no longer in the database.

This is by no means a complete record of everything that you can do with MongoDB, but it should be enough to get you started.  You can also refer to last week's post to get a few examples of interacting with nested values in MongoDB.  I hope that you will find today's post helpful in understanding how Mongo works, and that you will use it as a reference whenever you need it.  Happy querying!

Testing With Non-Relational Databases

Last week, I took a look at ways to query relational databases for testing.  This week I'm going to look at non-relational databases, describe how they are different from relational databases, and discuss how to query them in your testing.  Non-relational databases, such as MongoDB and DynamoDB, are sometimes called "NoSQL" databases, and are becoming increasingly popular in software applications.

The main difference between relational and non-relational databases is that relational databases use tables to store their data, where non-relational tables use documents.  The documents are often in JSON format.  Let's take a look at what the records in the Contacts table from last week's post would look like if they were in a non-relational database:

{

              contactId: "10000",

              firstName: "Prunella",

              lastName: "Prunewhip",

              email: "pprunewhip@fake.com",

              phone: "8005551000",

              city: "Phoenix",

              state: "AZ"

}

{

              contactId: "10001",

              firstName: "Joe",

              lastName: "Schmoe",

              email: "jschmoe@alsofake.com",

              state: "RI",

}

Note that Joe does not have a value for phone or city entered, so they are not included in his document.  This is different from relational databases, which are required to include a value for every field. Instead of having a NULL value for phone and city as Joe's record did in the SQL table, those fields are simply not listed.

Another key difference between relational and non-relational databases is that it's possible to add a new field into a table without adding it in for every document.  Let's imagine that we are adding a new record to the table, and we want that record to include a spouse's name.  When that record is added, it will look like this:

{

              contactId: "10002",

              firstName: "Amy",

              lastName: "Smith",

              email: "amysmith@faketoo.com",

              phone: "8885551001",

              city: "Boise",

              state: "ID",

              spouse: "John"

}

The original documents, 10000 and 10001, don't need to have this spouse value.  In a relational database if a new field is added, the entire schema of the table needs to be altered, and Prunella and Joe will need to have spouse values or NULL entered in for those fields.

With a non-relational database, it's not possible to do joins on table data as you saw in last week's post.  Each record should be treated as its own separate document, and you can do queries to retrieve the documents you want.  What that query language looks like depends on the type of the database used.  The examples below are using MongoDB's query language, which is JavaScript-based, and are querying on the documents listed above:

db.contacts.find() - this will return all the contacts in the table
db.contacts.find( { contactId: "10001" } ) - this will return the document for Joe Schmoe

To make the responses easier to read, you can append the command .pretty(), which will organize the data returned in JSON format rather than a single line of values. 

You can also run a query to return a single field for each document:

db.contacts.find({}, {firstName:1, _id:0}) - this will return just the first name for each contact

Because the documents in a non-relational database have a JSON-like structure, it's possible to have documents with arrays.  For example, our Contacts table could have a document that lists the contact's favorite foods:

{

              contactId: "10000",

              firstName: "Prunella",

              lastName: "Prunewhip",

              email: "pprunewhip@fake.com",

              phone: "8005551000",

              city: "Phoenix",

              state: "AZ",
              foods: [ "pizza", "ice cream" ]

}

It's even possible to have objects within arrays, as follows:

{

              contactId: "10001",

              firstName: "Joe",

              lastName: "Schmoe",

              email: "jschmoe@alsofake.com",

              state: "RI",
              pets: [ { type: "dog", name: "fido" }, { type: "cat", name: "fluffy" } ]

}

You can see how this type of data storage might be advantageous for your application's data.  Nesting data in this fashion makes it easier to read at a glance than it would be in a relational database, where the pets might be in their own separate table.

To run a query that will return all the contacts that have cats, you would simply request:

db.contacts.find( {"pets.type":"cat"} )

To run a query that will return all the contacts that have cats named Fluffy, you would request:

db.contacts.find( {$and: [{"pets.type":"cat"},{"pets.name":"fluffy"}]} )

These are just a few simple examples of how to query data with a non-relational database, and they should be enough to get you started in your testing.  To learn more, be sure to read the documentation for the type of database you are using.  As non-relational databases become increasingly popular, this knowledge will be extremely useful.