Email Subscription Form

Saturday, September 14, 2019

What I Learned at POST/CON Part I: Examples and Mocking

I've just returned from POST/CON, the annual Postman users' conference, and I am so excited about everything I learned there!  So excited, in fact, that I'm going to devote not one, but TWO blog posts to sharing my findings.

If you aren't already using Postman for your API testing, why on earth not?  It's the best API testing tool out there!  My opinion was reinforced this week, when I learned just how easy it is to create API examples and mock responses.  I'll be teaching you how to do both things in today's post.



The instructions in this post will be for the free version of Postman, and I'll be using version 7.  Your results will look slightly different in version 6; and the Pro version of Postman will have more functionality for the documentation and mocks than will be described here.

First, examples: why would you want to create an example of a request?  Because it's a great way to show other people on your team how the request is supposed to work.  It can also be used to create documentation for your API. 

The request I'll be using for today's post is one of the GET requests from Mark Winteringham's wonderful API, Restful-Booker.  Here is the request and the response:


You can see that this is a GET request that is asking for the booking with the ID of 1, and that the response returns the booking with the first and last name, checkin and checkout dates, and other details. 

If we wanted to teach someone else on our team how this request works, we could simply share this request with them.  But- what if the API is still being created and doesn't work yet?  Or what if our teammate doesn't have the correct authentication access to the request, and can't run it?  We can show them exactly what's supposed to happen with the request by creating an example.  In the upper right corner of the Postman window, just underneath the environment dropdown, is a link that says "Examples":

Click on this link, then click "Add Example".  A new request tab will open up with the same name as the original GET request, prefaced by "e.g":

The request will already have the HTTP verb, the URL, and any headers or request parameters set.  All you need to do to finish the example is to paste in an example of what the request response should be:
and set the appropriate response code:
Save the example, and return to the original GET request.  You'll see that now there is an example request listed in the top right:
Now anyone who sees the request can click on the Examples link to see exactly how the request is supposed to behave. 

Examples are also great for showing how an API request should behave in negative scenarios, such as when an id is not found, or when a user is not authenticated.

Another great feature of examples is that you can use them in your documentation.  You don't need to have examples to create documentation, but it makes your documentation much easier to understand.  To create documentation in Postman, you need a collection with requests.  Click on the three-dot menu beside the collection, and choose "Publish Docs":
A "Publish Collection" web page will open.  Select an environment if you like, and click the Publish Collection button.  Once published, you'll see a URL that you can go to view your documentation.  Go to that URL, and you'll see your request with your example response!
You can also use your example requests to create a mock server.  This can be used whenever you don't have access to the actual API server, such as when a feature is still being developed or when you are doing contract testing with another API. 

To set up a mock server, simply click on the three-dot menu beside the collection name and choose "Mock Collection".  You'll be presented with a pop-up window like this:
Give your mock collection a name, and click the "Create mock server" button.  You'll be assigned a special mock server that looks like this: https://<some guid>.mock.pstmn.io. 

This mock server is designed to return the response you created in your GET example whenever you make a request with an endpoint that matches the GET example.  Copy the URL for the mock server and paste it in a GET request, and then add the appropriate endpoint for your example: /booking/1.  Click send, and you should get this response:
Now you can save this request, naming it something like MOCK Get Booking, and you can save it to a collection called something like MOCK Restful Booker. 

You can create examples and mock requests like this for every request in an API, and when you have finished, you will have a complete documentation of your API as well as a mock server that will allow you call an API and get an appropriate response without actually connecting to the API's server!

I hope you find this helpful in your work with APIs.  Next week, I'll have more great knowledge from POST/CON to share!

Saturday, September 7, 2019

Your Test Cases Are Slowing You Down

One of the first QA jobs I had was a position at a company that made software that could be used to create mobile applications.  It was a very complex application, with so many features that it was often hard to keep track of them all.  Shortly before I started working there, the company had adopted a test tracking system to keep track of all of the possible manual tests the team might want to run.  This amounted to thousands of test cases.

Many of the test cases weren't written well, leaving those of us who were new to the team confused about how to execute them.  The solution to this problem was to assign everyone the task of revising the tests as they were run.  This helped a bit, but slowed us down tremendously.  Adding to the slowdown was the fact that every time we had a software release, our manager had to comb through all the tests and decide which ones should be run.  Then there was the added confusion of deciding which mobile devices should be used for each test.

We were trying to transition to an Agile development style, but the number of test cases and the amount of overhead needed to select, run, and update the tests meant that we just couldn't adapt to the pace of Agile testing.


You might be thinking at this point, "Why didn't they automate their testing?"  Keep in mind that this was back when mobile test automation was in its infancy.  One of our team had developed a prototype for an automated test framework, but we didn't have the resources to implement it because we were so busy trying to keep up with our gigantic manual test case library.

Even when you have a robust set of automated tests in place, you'll still want to do some manual testing.  Having a pair of eyes and hands on an application is a great way to discover odd behavior that you'll want to investigate further.  But trying to maintain a vast library of manual tests is so time consuming that you may find that you don't have time to do anything else!

In my opinion, the easiest and most efficient way to keep a record of what manual tests should be executed is through using simple spreadsheets.  If I were to go back in time to that mobile app company, I would toss out the test case management system and set up some spreadsheets.  I would have one smoke test spreadsheet; and one regression test spreadsheet for each major feature of the application.  Each time a new feature was added, I'd create a test plan on a spreadsheet, and once the feature was released, I'd either add a few test cases to a regression test spreadsheet (if the feature was minor), or I'd adapt my test plan into a new regression test spreadsheet for that feature.

This is probably a bit hard to imagine, so I'll illustrate with an example.  Let's say we have a mobile application called OrganizeIt!  Its major features are a To-Do List and a Calendar.  Currently the smoke test for the app looks like this:

TestiOS phoneiOS tabletAndroid phoneAndroid tablet
Log in with incorrect credentials
Log in with correct credentials
Add an event
Edit an event
Delete an event
Add a To-Do item
Edit a To-Do item
Complete a To-Do item
Mark a complete item as incomplete
Delete a To-Do item
Log out

And then we also have a regression test for the major features: Login, Calendar, and To-Do List.  Here's an example of what the regression test for the To-Do List might look like:

TestExpected result
Add an item to the list with too many charactersError message
Add an item to the list with invalid charactersError message
Add a blank item to the listError message
Add an item to the list with a correct number of valid charactersItem is added
Close and reopen the applicationItem still exists
Edit the item with too many charactersError message, and original item still exists
Edit the item with invalid charactersError message, and original item still exists
Edit the item so it is blankError message, and original item still exists
Mark an item as completedItem appears checked off
Close and reopen the applicationItem still appears checked off
Mark a completed item as completed againNo change
Mark a completed item as incompleteItem appears unchecked
Mark an incomplete item as incomplete againNo change
Close and reopen the applicationItem still appears unchecked
Delete the itemItem disappears
Close and reopen the applicationItem is still gone

This test would also be run on a variety of devices, but I've left that off the chart to make it more readable in this post.

Now let's imagine that our developers have created a new feature for the To-Do List, which is that items on the list can now be marked as Important, and Important items will move to the top of the list.  In the interest of simplicity, let's not worry about the order of the items other than the fact that the Important items will be on the top of the list.  We'll want to create a test plan for that feature, and it might look like this:

TestExpected result
Item at the top of the list is marked ImportantItem is now in bold, and remains at the top of the list
Close and reopen the applicationThe item is still in bold and on the top of the list
Item at the middle of the list is marked ImportantItem is now in bold, and moves to the top of the list
Item at the bottom of the list is marked ImportantItem is now in bold, and moves to the top of the list
Close and reopen the applicationAll important items are still in bold and at the top of the list
Every item in the list is marked ImportantAll items are in bold
Close and reopen the applicationAll items are still in bold
Item at the top of the list is marked as normalThe item returns to plain text, and moves below the Important items
Close and reopen the applicationThe item is still in plain text, and below the Important items
Item in the middle of the Important list is marked as normalThe item returns to plain text and moves below the Important items
Item at the bottom of Important list is marked as normalThe item returns to plain text and is below the Important items
Close and reopen the applicationAll important items are still in bold, and normal items are still in plain text
Delete an important itemItem is deleted
Close and reopen the applicationItem is still gone
Add an item and mark it as importantThe item is added as important, and is added to the top of the list
Add an item and mark it as normalThe item is added as normal, and is added to the bottom of the list
Close and reopen the applicationThe added items appear correctly in the list
Mark an important item as completedThe item is checked, and remains in bold and at the top of the list
Close and reopen the applicationThe item remains checked, in bold, and at the top of the list
Mark an important completed item as incompleteThe item is unchecked, and remains in bold and at the top of the list

We would again test this on a variety of devices, but I've left that off the chart to save space.  

Once the feature is released, we won't need to test it as extensively, unless there's some change to the feature.  So we can add a few test cases to our To-Do List regression test, like this:

TestExpected result
Add an item to the list with too many charactersError message
Add an item to the list with invalid charactersError message
Add a blank item to the listError message
Add an item to the list with a correct number of valid charactersItem is added
Close and reopen the applicationItem still exists
Add an important item to the listItem is in bold, and is added to the top of the list
Edit the item with too many charactersError message, and original item still exists
Edit the item with invalid charactersError message, and original item still exists
Edit the item so it is blankError message, and original item still exists
Mark an important item as normalItem returns to plain text and is moved to the bottom of the list
Mark an item as completedItem appears checked off
Mark an important item as completedItem remains in bold text and appears checked off
Close and reopen the applicationItem still appears checked off
Mark a completed item as completed againNo change
Mark a completed item as incompleteItem appears unchecked
Mark an incomplete item as incomplete againNo change
Close and reopen the applicationItem still appears unchecked
Delete the itemItem disappears
Close and reopen the applicationItem is still gone
Delete an important itemItem disappears

The new test cases are marked in red, but they wouldn't be in the actual test plan.  

Finally, we'd want to add one test to the smoke test to check for this new functionality:

TestiOS phoneiOS tabletAndroid phoneAndroid tablet
Log in with incorrect credentials
Log in with correct credentials
Add an event
Edit an event
Delete an event
Add a To-Do item
Add an important To-Do item
Edit a To-Do item
Complete a To-Do item
Mark a complete item as incomplete
Delete a To-Do item
Log out

With spreadsheets like these, you can see how it is easy to keep track of a huge amount of tests in a small amount of space.  Adding or removing tests is also easy, because it's just a matter of adding or removing a line to the table.  

Spreadsheets like this can be shared among a team, using a product like Google Sheets or Confluence.  Each time a smoke or regression test needs to be run, the test can be copied and named with a new date or release number (for example, "1.5 Release" or "September 2019"), and the individual tests can be divided among the test team.  For example, each team member could do a complete test pass with a different mobile device.  Passing tests can be marked with a check mark or filled in green, and failing tests can be marked with an X or filled in red.

And there you have it!  An easy to read, easy to maintain manual test case management system.  Instead of taking hours of time maintaining test cases, you can use your time to automate most of your tests, freeing up even more time for manual exploratory testing.  

Saturday, August 31, 2019

Break Your App With This One Weird Trick

I missed a bug recently, more than once, and I'm kicking myself about it.  This post is about that bug and how you should make sure to always run a test for it. It's also about how to keep from repeating your mistakes.

Here's what happened: as I mentioned in a previous post, I'm currently testing file uploads and downloads.  When the developers on my team first coded the upload functionality, I dutifully tested all kinds of file names: long names, short names, names without extensions, names with capital letters, etc.  Everything looked great, but I missed one important test: testing file names with spaces.  Every file name I had tested with was one single word, like "sunrise.jpg".  I forgot to test with a name like "Grand Canyon.jpg", and as it turned out, uploads with spaces in them weren't working correctly.



Spaces are SO easy to forget to test, because they are literally invisible!  Testing with spaces applies to any text fields, not just file names.  For example, when you are testing a first name field, make sure to test with two first names, like "Mary Jo".

It's also important to remember to test with a space at the beginning of the text and at the end of the text.  Your developer should be trimming these whitespaces when processing input, but if he or she forgets, this can mean trouble.  You may wind up with a situation where you have a list of names sorted by last name, and the name "Smith" is appearing at the top.  This is probably because someone entered " Smith" with a leading space.

Similarly, your users might have problems logging in to your application, even though they are sure they have the right username.  This may be because when they set their username, they accidentally put a space at the end of the username, and that space was not trimmed by the developer.  So they are trying to log in with "catLover" when they should really be logging in with "catLover ".

Back to my story: after the bug with file uploads was discovered and fixed, I carefully retested uploads again, this time being sure to include file names with spaces in the beginning, middle, and end of the file.

Our next development task was to be able to resize a file upon request.  When this functionality was ready, I started running all kinds of tests: resizing by height only, width only, height and width, resizing various file types, etc.  While I was testing, the developer who worked on the feature mentioned that he had just discovered a bug: the files wouldn't resize if they had spaces in the names, because the spaces weren't being encoded properly.  I'd like to think I would have discovered this eventually, but who knows?  I was more focused on the new functionality than I was on regression testing.

The bug was fixed, and again I carefully tested it.  I added spaces in the beginning, middle, and end of the file name, and I used every kind of special character on my keyboard.  Surely, I thought, this must be the end of this bug.

Our next development task was to do file size and type checking.  We didn't want to upload a file if it was larger than the application was told to expect, or if it had the wrong file type.  I tested with files with all kinds of sizes and types, and all kinds of mismatches.  With each mismatch, I verified that the file was checked and rejected.  I did a great deal of regression testing as well.  It occurred to me to test with different file names, but since this I had already verified that file names of all kinds were working with resizing, it seemed like overkill, so I chose not to do it.

I was wrong!  As it turned out, the system that was doing the file checking was different from the system that was doing the file resizing, and once again, spaces in the file name weren't being encoded properly.  A developer noticed that every file that had spaces in the name wasn't being checked by the system.  I was bitten by the file space bug once again.

I hate making mistakes, and I hate missing bugs!  Fortunately these bugs were caught before they impacted any end users.  But I like to learn from my mistakes, and I've taken the following steps to make sure I don't miss this bug ever again:

  • I've put a sticky note that says "Spaces" on my desk.  Whenever I see it, I will be reminded to test for spaces in my inputs.  I did this years ago when I kept forgetting to test on different browsers.  I had a "Test on IE" note on my monitor, and it helped me develop the habit.
  • I've added file names with spaces to my saved Postman collections.  This way, even if I forget to test with spaces, the saved requests will remember for me.
  • I've written this blog post to serve as a cautionary tale to myself and others.

One final lesson I've learned from this is that different parts of the same feature can process information differently.  The feature is using new technologies that I've never tested before, and I assumed file names would be handled in the same way for uploads, resizes, and type checking. But that was not the case.  When you are dealing with new technologies, be sure to regression test everything, even things you think might not need to be covered.

It's amazing how much one simple space can break!  When you are testing anything that can accept text, from a simple form field to a file upload, be sure to remember those invisible spaces.

Saturday, August 24, 2019

Let's Go Deep! Part III: Internet Routing

In the last two posts, we've been going deep, learning about how information is transmitted over the Internet.  In Part I, we learned how data is divided into packets, sent to its destination and reassembled.  In Part II, we learned how data sent over the Internet can be encrypted and protected.  But how does data know how to get from one IP address to another?  This is the subject of today's post.


Every device that can be connected to the Internet has a network interface card (NIC), which is a piece of hardware installed on the device that allows it to make a network connection.  Each NIC has a media access control (MAC) address that is specific to only that device.

A modem is a device that connects to the Internet.  Other devices can connect to the modem in order to receive Internet transmissions.  A wireless router is capable of receiving wifi transmissions.  The router connects to the modem, and other devices connect wirelessly to the router in order to receive data from the Internet.  Many Internet service providers (ISPs) provide customers with a combination modem/router, which connects to the Internet and sends and receives wireless signals.

In Part I, we learned that every device connected to the Internet has an IP address.  An IP address is different from a MAC address in that the MAC address is assigned by the manufacturer of the device, and an IP address is assigned by the network.  An IP address has two sections, called the subnet and the host.  The subnet refers to one subsection of the entire Internet.  The host is the unique identifier for the device on the network.  The IP address combines these two numeric values using bitwise operations.  You can't look at an IP address and say that the numbers on the left make up the subnet and the numbers on the right make up the host; it doesn't work that way.  The IP address is more like a sum of two long numbers.

As mentioned in Part I, when a packet of data is sent from a server to a client, it is sent with the IP address of the destination.  In order to get to that destination, the packet will hop from one network to another.  The routing protocol of the Internet is called the Border Gateway Protocol (BGP).  This is a system that helps determine a route that will traverse the least number of networks to get to the destination.  Every router in a network has a series of routing tables, which are sets of directions for how to get from one network to another.

When a packet of information is first sent to the network's router, it looks at the IP address of the destination and determines if the directions to the destination are available in the routing tables.  If they are not, the BGP is used to determine the next logical network where the packet should be sent.  A gateway is an entrance to a network, and a default gateway is the address that the request is sent to if there's no knowledge of a specific address in that network.  When a packet arrives at the new gateway, the BGP calculates the next appropriate destination.

After traversing through networks in this way, eventually the packet arrives at the router for the network that contains the IP address of the destination.  The router determines the MAC address of the destination and sends the packet to that address.

One more important feature of networking is the use of a proxy server.  A proxy server is a server that is positioned between the client and the destination server.  It is configured so that any requests your client makes will go through the server before it gets to its destination.  There are many uses for a proxy server; the main use is to keep the actual address of a site or a router private.  Proxy servers can also be used by hackers to intercept requests, especially on a public network.  Finally, proxy servers are a great way to do security testing!  Using a tool like Fiddler or Burp Suite, you can intercept the requests that you make to your application and the responses you receive in return.  You can learn more about how to use Burp Suite in this post.

This concludes my three-part series on how the Internet works.  I hope that you have found it helpful, and that you can apply these concepts when testing!

Saturday, August 17, 2019

Let's Go Deep! Part II: Encryption, Tokens, and Cookies

In last week's post, we talked about how HTTP works to pass information from a server to a browser.  But when information is passed back and forth between systems, we need to make sure that it's protected from being intercepted by others for whom it was not intended.  That's why HTTPS was created.  In this week's post, we'll talk about how encryption is used in HTTPS, what the difference is between cookies and tokens, the different types of cookies, and how cookies can be protected.


How HTTPS Works:

When two systems communicate with each other, we refer to them as the client and the server.  The client is the system making the request, such as a browser, an application, or a mobile device, and the server is the system that supplies the information, such as a datastore.  HTTPS is a method of securely transmitting information between the client and the server.  HTTPS uses SSL and TLS to encrypt the data being transmitted and decrypt it only when it arrives at its destination.  SSL (Secure Sockets Layer) and TLS (Transport Layer Security) are both tools for the encryption and decryption of data; TLS is a newer version of SSL.  

Here's how TLS works: before any data is transmitted, the client and the server first perform a handshake.  The handshake begins with the client contacting the server with a suggested encryption method and the server responding back agreeing to use that encryption method.  It then continues with the client and the server swapping certificates.  A certificate is like an ID card; it verifies the identity of the client or server.  The certificates are checked against the CA (Certificate Authority), which is a third-party entity that creates certificates specifically for the purpose of HTTPS communication.  

Once the certificates are swapped and verified, the client and the server swap decryption keys, and the handshake is completed.  Now the data is encrypted, transmitted from the server to the client, and decrypted once it arrives at the client safely.  

Session Cookies and Tokens:

Another important way data is secured is through the use of session cookies or tokens.  Session cookies and tokens are strings that are passed in with a client's request to verify that the person making the request has the right to see the data requested.  The main difference between session cookies and tokens is that a session cookie is stored both on the client and the server, and a token is only stored on the client.  

In systems that use tokens, the token is created when a user logs in.  The token is made up of encrypted information that identifies the user.  The token is stored in local storage in the client's browser and is sent with every HTTPS request to the server.  When the server receives the token, it decrypts it, validates the user's information, and then returns the response.  

The most popular system of tokens in use today is JWT (JSON Web Token).  You can read more about JWTs in this helpful documentation.  

A session cookie is a unique string that is created by the server when the user logs in.  It is saved in the server's datastore as a session id.  The server returns the cookie to the client, and the client saves it in the browser while the session is active.  Whenever the client makes a request to the server, it sends the cookie with the request.  The server then compares the cookie with the one it has saved to make sure it matches before returning the response.  

Tokens and session cookies are usually set to expire after a period of time or after an event.  For example, a token issued might be good for one hour.  Just before the hour is up, a request can be made for a new token (called a refresh token) in order to keep the user signed in.  Session cookies usually expire when the user logs out or when the browser is closed.

Persistent Cookies:

Another type of cookie used is the persistent cookie, which is a bit of data saved on the server about the user's preferences.  For example, if a user goes to a website and chooses German as the language they would like on the site, a persistent cookie will remember that information.  The next time the user goes to the site, the cookie will be examined and the site will load in German.  

Securing Cookies:

Because they are stored on the server, cookies are more vulnerable to being intercepted and used by someone other than the user than tokens are.  To help protect cookies, these flags (attributes) can be added to them at the time of creation:

  • Secure flag: ensures that the cookie can only be transmitted over HTTPS requests and not over HTTP requests.
  • HttpOnly flag: keeps a cookie from being accessed via JavaScript, which helps protect it from Cross-Site Scripting (XSS) attacks.  
  • SameSite flag: ensures that the cookie can only be sent from the original client that requested the cookie, which helps protect it from Cross-Site Request Forgery attacks.  


Hopefully this post has helped you learn how HTTPS works, and how tokens and cookies ensure a client's validity.  Researching this information certainly helped me!  But I want to go deeper: how does a message know how to get from one IP address to another?  How are HTTP requests intercepted?  And how does a router work?  I'll have answers to these questions in next week's post!  


What I Learned at POST/CON Part I: Examples and Mocking

I've just returned from POST/CON, the annual Postman users' conference, and I am so excited about everything I learned there!  So ex...