Easy Free Automation Part VI: Security Tests

Often when people think of security testing, they think of complicated software scans, request intercepts, and IP address spoofing.  But some of the most crucial application security testing can be done simply through making API requests.  In this week's post, I'm taking a look at examples of authentication testing, authorization testing, and field validation testing.

As I have in every post in this Easy Free Automation series, I've created an example that you can download here.  This is a simple json file that can be run with Newman.  As you recall from Easy Free Automation Part III: Services Tests, Newman is the command-line runner for Postman, my favorite API testing tool.  If you need to install Postman or Newman, take a look at that post.

For my test application, I'm using the awesome Restful-Booker API.  It's worth noting that this API does come with some intentional bugs, two of which I'll mention below.

The json file I've made available on Github is for the test collection.  I didn't include an environment file this week, because I didn't need to store any variables.  Once you have downloaded the json file, open a command window, change directories to get to the directory where the file is stored, and type newman run easyFreeSecurityTests.json. You should see sixteen tests run and pass.

Let's take a look at the kinds of tests we're running.  The tests will be easier to interpret if you upload them into Postman; take a look at this post if you need help doing that.

The first six tests in the collection are authentication tests.  I am verifying that I can't log in with invalid credentials.  But I'm verifying six different invalid username and password combinations:

• Create token with empty username
• Create token with invalid username
• Create token with empty password
• Create token with invalid password
• Create token with empty username and password
• Create token with invalid username and password

This may seem like overkill, but I have actually encountered bugs where a user can log in if the password field is blank, and where a user can log in if both the username and password are incorrect. 

The assertion I am using for each of the six authentication tests is the following:
pm.test("Bad credential message returned", function () {
    pm.expect(pm.response.text()).to.include("Bad credentials");
});

Ordinarily I would assert that the response code I was getting was a 401, but since this request is (incorrectly) returning a 200, I'm instead verifying the text of the response: "Bad credentials".  (For more information on HTTP error codes, see this post.)

The next six tests in my collection are authorization tests.  There are three actions in the Restful-Booker that require a valid token: Put, Patch, and Delete.  So for each of these requests, I'm testing that I cannot run the request with a missing token, and I cannot run the request with an invalid token:

• Update booking with no token
• Update booking with invalid token
• Partial update booking with no token
• Partial update booking with invalid token
• Delete booking with no token
• Delete booking with invalid token

For each of these requests, I am asserting that I receive a 403 status code as a response:

pm.test("Status code is 403", function () {

     pm.response.to.have.status(403);

});

If a developer makes a change to the code and accidentally removes the token requirement for one of these operations, automated tests like these will discover the error right away, because the response code will change to a 200 or a 201.

Finally, I have four field validation tests.  I would like to have more tests here, but because some of the fields in this API aren't validated, I'm sticking to the date fields.  In each of these tests, I am sending in an invalid date:

• Create booking invalid checkin month
• Create booking invalid checkin day
• Create booking invalid checkout month
• Create booking invalid checkout day

In each of these tests, I am validating that I receive an invalid date message:

pm.test("Invalid date response", function () {

     pm.response.to.have.body("Invalid date");

});

Field validation might not seem like a security concern, but it's one of the easiest ways to hack an application, through entering in a script for XSS or a SQL command for SQL injection.  By verifying that the application's input fields are only allowing in certain data types or formats and only allowing in a certain number of characters, we are going a long way towards protecting ourselves from these attacks.

Astute readers will have noticed that we could also have date field validation on the PUT requests that update a booking, and the PATCH requests that partially update a booking.  And of course, if the other fields such as First Name and Last Name had validation (as they should), we would want to test that validation as well.

Running simple repetitive tests like this is not particularly glamorous and will never make headlines.  But it's simple tests like these that can catch a security problem way before it becomes an issue.  Just recently I was alerted that some tests I had set up to run nightly in a QA environment were failing.  Upon investigation, I discovered that my test user was now able to access information belonging to another user.  If I hadn't had authorization tests in place, that security hole might have been missed and might have made it to production.  

Next week, we'll move on to Easy Free Load Tests!

Easy Free Automation Part V: Visual Tests

Visual tests are more than just UI tests; they verify that what you are expecting to see in a browser window is actually rendered.  Traditional UI tests might verify that a page element exists or that it can be clicked on, but they don't validate what the element looks like.  Fortunately, there are a variety of ways to do visual validation.  My favorite way is through Applitools, but since this series of posts is on "Easy Free Automation", I needed to look elsewhere for visual validation this week.

I settled on a fun tool called BackstopJS.  It was pretty easy to set up, and while I haven't yet discovered everything that it can do, I created a simple example with two tests that you can clone or download here.

Once you have cloned or downloaded these files, it's time to install BackstopJS.  I'm assuming that you already have Node and npm installed; if you don't, see last week's post for instructions.  To install BackstopJS, simply open your command window and type npm install -g backstop.js.  

After the installation completes, navigate using the command line to the folder where you have downloaded the easyFreeVisualTests files, and type backstop test.  This will run the visual tests for the first time, and when the tests complete, a browser window will pop up with the test results.  You'll notice that the tests all failed; this is expected, and we'll fix this in a moment.  Before we do, take a look at the test results.  You should see four tests: two tests with screenshots of Google's homepage, and two tests with screenshots of my picture (taken from my personal website).  The tests failed because at the moment, there is nothing to compare the screenshots to.  We can accept the screenshots and use them for future comparisons by typing backstop approve in the command line.  Now run backstop test again, and you will see the four tests run and pass.

Let's take a look at the backstop.json file to see how these tests are configured.  Near the top of the file, we see the viewports section:

This section is showing what screen sizes we want to test with.  In this case, we are testing with a phone screen size and a laptop browser size.

A little farther down, we see the scenarios section.  The first scenario is for the Google Search Homepage:

We see the label of the test, along with the url.  Also of note is the "misMatchThreshold".  Setting this to a higher number means that the test will allow for a higher percentage of pixel mismatches before failing the test.

The second scenario is for my personal webpage:

Note that this scenario is using the "selectors" section.  In this section, I'm specifying the specific element I want to look at: my picture.  When the test runs, rather than taking a screenshot of the entire page, it takes a screenshot of this single element and compares it to the baseline screenshot.  This is great for pages that have content that changes frequently; you can set your test to look only at the elements that stay the same.

If you want to add BackstopJS to your own Javascript project, it's easy to do!  Simply navigate to your project folder in the command line and run backstop init. This will add a backstop.js file to your existing project, and you can configure it as needed.

It would certainly be possible to have both Backstop and other UI automated tests installed in the same project.  When your automation runs, you could run through all of your UI tests first, then run your Backstop tests to verify all your visual elements are appearing as expected.

Next week, we'll move on to automated security tests!

Easy Free Automation Part IV: UI Tests

I'll be honest: UI tests are my least favorite automated tests.  This is because they are often so hard to set up.  There are dozens of different ways to run automated UI tests, but this can make things more confusing because it's hard for someone new to automation to figure out what to do.

So when I prepared to write this week's post, my primary goal was to make it as easy as possible to get started with UI testing.  And of course, I also wanted the framework to be free, with no need to purchase a tool.

I've arrived at a way to get up and running with UI automation using Node and Selenium Webdriver in just six steps.  While I have only tested this process on two computers, I believe these steps will be effective for most people.  The one prerequisite is that you need to have Chrome installed, because that's the browser that we will be using for the test.

Setting Up Automated UI Testing in Six Easy Steps:

1. Open up a command window and verify that you have Node.js installed by typing
node --version
If you get a version number in response, you have Node installed.  If it's not installed, you can install it here: https://nodejs.org/en/.

2. If you needed to install Node in Step 1, add Node to your system's PATH (see instructions for Windows here and instruction for Mac here).  After you've done this, reboot your computer so the new PATH will be recognized.  Check one more time that Node is installed by typing
node --version again.

3. When you install Node, the npm package manager should be installed automatically.  Verify that npm is installed by typing
npm --version
If you get a version number in response, npm has been installed.  If not, you can check this link for instructions about installing npm.

4. Open a browser and go to this GitHub repo.   If you have Git installed, you can clone the project.  If not, you can download a zipfile of the project and extract it.

5. In the command window, navigate to the folder where the project has been installed.  The project folder should contain a test.js file and a package.json file.  (See instructions here about navigating in the command line.)  Type this command:
npm install
This will install everything you need to run the automated test.

6. Type this command:
node test
This will run the test.js file, which has one test.  You should see an instance of Chrome browser open, run the test, and close again!

Let's take a look at what the test.js file does:

var webdriver = require('selenium-webdriver'),
    By = webdriver.By,
    until = webdriver.until;
var chrome = require('selenium-webdriver/chrome');
var path = require('chromedriver').path;

var service = new chrome.ServiceBuilder(path).build();
chrome.setDefaultService(service);

var driver = new webdriver.Builder()
    .withCapabilities(webdriver.Capabilities.chrome())
    .build(); 

The purpose of all of this code is to require webdriver and chrome driver, setting up the Chrome driver, and to set up the "By" and "until" classes, which are helpful for automated UI testing.

driver.get('http://www.google.com');

This command tells the driver to navigate to the Google home page.

driver.findElement(By.name('q')).sendKeys('webdriver\n');

This command tells the driver to find the web element named "q", which is the search box; type "webdriver" into the search box; and click the Return key.

driver.findElement(By.partialLinkText("seleniumhq")).click();

This command looks through the search responses for the element that has "seleniumhq" in the link text, and once the element has been found, it clicks on it.

driver.wait(until.elementLocated(By.id('sidebar')));

This is waiting for the Selenium Webdriver page to load by watching for the element with the id called 'sidebar'.

driver.getTitle().then(function(title) {
    if(title === 'Selenium WebDriver') {
      console.log('Test passed');
    } else {
      console.log('Test failed');
    }
    driver.quit();
});

Once the element has been located, then the driver looks at the title of the page and checks to see if it is what was expected.  If the title matches "Selenium Webdriver", it logs to the console that the test passed, and if it does not match, it logs to the console that the test failed.  Finally, the driver closes the browser window.

Hopefully this post has helped you with the most difficult part of automated UI testing- the setup!  Once you are up and running, there are lots of great tutorials that describe how to locate and interact with browser elements using Webdriver.  You can find the documentation for the Webdriver By class here, and I have some old blog posts about element locators here, here, and here.   The posts were written for Java, but the same concepts apply when you are locating elements in Node.

The most important thing to remember about automated UI testing is that it should be done sparingly!  Whatever you can test with unit and services tests should be tested that way instead.  UI testing is best for validating that elements are on a web page, and for running through simple user workflows.  Next week, we'll go on to an important addition to UI testing: visual testing.

UPDATE: If you are experiencing an issue where you get an "unhandled promise rejection", try running this command:  npm install selenium-webdriver@3.6.0 and then try running the test again.

Easy Free Automation Part III: Services Tests

This week we are looking at my favorite type of automated tests: services tests.  The reason I love services tests is because they test so much of the application without the hassle of the UI.  There are many types of services, but the most widely used service is the REST API.  And my favorite way to test a REST API is with Postman. 

I'm not going to get into too many details about REST verbs or about Postman, because I have already done so in earlier blog posts.  If you'd like to catch up on what I've written, you can start with this post and work your way forward week by week.  You can also read this tutorial that I wrote for Simple Programmer.  

This post will focus on how you can automate Services Tests.  To demonstrate, I've created a collection of Postman tests that you can download from Github here.  The API I'm testing is called Restful-Booker: it's a great easy API created by Mark Winteringham to help people learn how to test APIs.  

Once you have downloaded the two files from Github, you can upload them in Postman to see what they do.  To upload the collection, click on the Import button on the top left of the Postman window:

When the Import window pops up, click the Choose Files button and then navigate to the location of the file you downloaded called restfulbooker.collection.json.  Click on that file name, and the collection will be imported:  

Next, you'll need to upload the environment file.  The environment file is what contains the variables that are used in the requests and assertions.  To upload the environment file, click on the gear icon in the top right of the Postman window:

When the environment window pops up, click on the Import button, then on the Choose Files button, then navigate to the location of the downloaded file called restfulbooker.env.json.  Click on that file name and the environment will be imported: 

Finally, in the dropdown menu in the top right, select the Restful Booker environment: 

You are now ready to run the requests from within Postman.  Keep in mind that the tests are not idempotent: some of the requests depend on previous requests for setting variables and creating bookings, so while you are exploring you should run the requests in order.

Let's take a look at one of the requests and its assertions:

This is a GET request.  The URL of the request is https://restful-booker.herokuapp.com/booking/1.  The request is asking for the booking with the id of 1.  You can see two tests in the Tests section: the first test verifies that the response code of the request is 200, and the second test verifies that "firstname" is one of the fields in the response.  

Click the Send button to run the request, then scroll down to the bottom of the page:

Here you see the body of the response, which has returned a booking.  You can also see a tab called Test Results, which shows that 2 out of 2 tests pass.  If you click on this tab, you can see your results:

Now that you understand how the tests and assertions work, let's run them from the command line!  In order to do this, you will need to have node.js installed.  Installing node will also install npm, which is the node package manager.  Then you can use npm to install Newman, the package that is used to run Postman tests.  To install Newman, simply open your command-line window and type: 

npm install -g newman

Once Newman is installed, navigate in the command-line window to the place where your restfulbooker.collection.json and restfulbooker.environment.json files are located.  Now you can run your tests with this command::

newman run restfulbooker.collection.json -e restfulbooker.environment.json

(This command should be in a line with no returns; it's OK if it wraps around to a second line, though.)

The "-e" in this command stands for "environment".  If all goes well, you should get a result that looks like this:  

You can also run your tests from other file locations.  To do this, simply specify the path to your files in your command.  For example, if my files were in a folder called NightlyTests, which was in my Documents folder, and if I were using a Windows computer, I would use this command:

newman run C:\Users\KJackvony\Documents\NightlyTests\restfulbooker.collection.json -e C:\Users\KJackvony\Documents\NightlyTests\restfulbooker.environment.json

Now that you know how to run your tests from the command line, you can set up your tests to run automatically with a cron job or in a CI/CD platform such as Jenkins.  Just be aware that whatever machine you run your tests on will need to have Newman installed.  

You can also have your test results write to a file with the -r command.  For example, you can have your results written in junit with this command:

newman run restfulbooker.collection.json -e restfulbooker.environment.json -r junit

When the tests are finished running, a Newman folder will be created with your test results inside.

I hope that this tutorial has given you some ideas about how you can automate your API testing!  Next week we'll move on to UI tests with Selenium.