New Blog Location!

I've moved!  I've really enjoyed using Blogger for my blog, but it didn't integrate with my website in the way I wanted.  So I've moved my blog to WordPress, which is now also hosting my site.

When you navigate to www.thinkingtester.com, you'll see not only my blog posts, but also my latest articles, videos, courses, and other free goodies.

The posts on this blog will continue to exist, so your old links will still work; I just won't be adding new content to this page. I'm looking forward to seeing you at Think Like a Tester!

Book Review: Unit Testing Principles, Practices, and Patterns

It's book review time once again, and this month I read Unit Testing Principles, Practices, and Patterns by Vladimir Khorikov.  I thought that a book about unit testing would be pretty dry, but it was really interesting!

Since I am not a developer I don't usually write unit tests, although I have done so occasionally when a developer asks me to help.  Being a good tester, I knew to do things like mock out dependencies and keep my tests idempotent, but through this book I discovered lots of things I didn't know about unit testing.

The author has a background in mathematics, and it shows.  He is very systematic in his process of explaining good unit test patterns, and each chapter builds upon the previous one.  Here are some of the important things I learned from this book:

• There are two schools of thought about unit testing: the classical school and the London school.  In the classical school, unit tests are not always limited to a single class.  The tests are more concerned with units of behavior.  Dependencies, such as other classes, don't need to be mocked if they aren't shared.  In the London school, unit tests are limited to a single class, and calls to other classes are always mocked, even if they are part of the same code base and not shared with any other code.

• Unit tests should always follow this pattern:
• Arrange- where the variables, mocks, and system under test (SUT) are set up
• Act- where something is done to the SUT
• Assert- where we assert that the result is what we expect

• The Act section of the unit test should only have one line of code.  If it has more than one line of code, that probably means that we are testing more than one thing at a time.

• A good unit test has the following characteristics:
• It's protected against regressions- it shouldn't break when you change something unrelated in the code
• It's resistant to refactoring- refactoring the code shouldn't break the test
• It provides fast feedback
• It's maintainable- it's easy for someone to look at the test, see what it's supposed to do, and make changes to it when needed

• Mocks and stubs are both types of test doubles: faked dependencies in tests which are used instead of calling the real dependencies in order to keep the tests fast, resilient, and focused only on the code being tested.
• Mocks emulate outgoing interactions, such as putting a message on a service bus
• Stubs emulate incoming interactions, such as receiving data from a database

• Test doubles should only be used with inter-system communications: calls to something outside the code, like a shared database or an email server.  For intra-system communications, where a datastore or class is solely owned by the code, the call shouldn't be mocked or stubbed.

The most interesting thing I learned from the book was that it's really hard to write good unit tests when the code is bad.  The author provides lots of examples of how to refactor code in order to make tests more robust.  These practices also result in better code!  Reading through the examples, I now understand how to better organize my code by separating it into two groups: code that makes a decision, such as function that adds two numbers, and code that acts upon a decision, such as writing a sum to a database.

The author doesn't just write about unit tests in this book; he also describes how to write integration tests, and provides examples of writing tests for interacting with databases.  

I learned much more than I was expecting to from this book!  Software test engineers will find many helpful ideas for all types of automation code in this book. Software developers will not only improve their unit test writing, but also their coding skills.  I recommend it to anyone who would like to improve their test automation.

Managing Test Data

It's never fun to start your work day and discover that some or all of your nightly automated tests failed. It's especially frustrating when you discover that the reason why your tests failed was because someone changed your test data. 

Test data issues are a very common source of aggravation for software testers. Whether you are testing manually or running automation, if you think your data is set the way you want it, and it has been changed, you will waste time trying to figure out why your test results aren't right. 

Here are some of the common problems with test data:

Users overwrite each other's data

I was on a team that had an API I'll call API 1. I wrote several automated tests for this API using a test user. API 1 was moved to another team, and my team started working on API 2. I wrote several automated tests for API 2 as well. Unfortunately, I used the same test user for API 2, and this test user needed to have a different email address for API 2 than it did for API 1. This meant that whenever automated tests were run on API 1, it changed the address of the test user, and then my API 2 tests would fail.

Configuration is changed by another team

When teams need to share a test environment, changes to the environment configuration made by one team can impact another team. This is especially common when using feature toggles. One team might have test automation set up with the assumption that a feature toggle will be on, but another team might have automation set up with the expectation that the feature toggle is off.

Data is deleted or changed by a database refresh

Companies that use sensitive data often need to periodically scramble or overwrite that data to make sure that no one is testing with real customer information. When this happens, test users that have been set up for automation or manual testing can be renamed, changed, or deleted, causing tests to fail.

Data becomes stale

Sometimes data that is valid at one point in time becomes invalid as time passes. A great example of this is a calendar date. If an automated test needs a date in the future, the test writer might choose a date a year or two from now. Unfortunately, in a year or two, that future date will become a past date, and then the test will fail.

What can we do to better manage test data? Here are some suggestions:

Use Docker

Using a virtual environment like Docker means that you have complete control over your test environment, including your application configuration and your database. To run your tests, you spin up a virtual machine, run the tests, and destroy the machine when the tests have completed.

Create a fresh database for testing

It's possible to create a brand-new database for the sole purpose of running your test automation. With Windows, this can be accomplished by creating a SQL DACPAC. You can set your database schema, add in only the data that you need for testing, create the database, point your tests to that database, and destroy the database when you are finished.

Give each team their own test space

Even if teams have to share the same test environment, they might be able to divide their testing up by account. For example, if your application has several test companies, each team can get a different test company to use for testing. This is especially helpful when dealing with toggles; one team's test company can have a feature toggled on while another team's test company has that feature toggled off.

Give each team their own users

If you have a situation where all teams have to use the same test environment and the same test account, you can still assign each team a different set of test users. This way teams won't accidentally overwrite one another's data. You can give your users names specific to your team, such as "Sue GreenTeamUser". 

Create new data each time you test

One great way to manage test data is to create the data you need at the beginning of the test. For example, if you need a customer for your test, you create the new customer at the beginning of your test suite, use that customer for your tests, and then delete the customer at the end of your tests. This ensures that your test data is always exactly the way you want it, and it doesn't add bloat to the existing database.

Use "today+1" for dates in the future

Rather than choosing an arbitrary date in the future, you can get today's date, and then use an operation like DateAdd to add some interval, like a day, month, or year, to today's date. This way your test date will always be in the future. 

Working with test data can be very frustrating. But with some planning and strategy, you can ensure that your data will be correct whenever you run a test. 

Why We Test

Most software testers, when asked why they enjoy testing, will say things like:

• I like to think about all the ways I can test new features
• It's fun to come up with ways to break the software
• I like the challenge of learning how the different parts of an application work

I certainly agree with all of those statements!  Testing software is fun, creative, and challenging. 

But this is not WHY we are testing.  We test to find out things about an application in order to ensure that our end users have a good experience with it.  Software is built in order to be used for something; if it doesn't work well or correctly, it is not accomplishing its purpose.  

For example:

• If a mobile app won't load quickly, users will stop using it or delete the app from their phone
• If a financial app has a security breach, they'll lose customers and may even be sued for damages
• If an online store has a bug that keeps shoppers from completing their purchases, the company will lose out on sales

There are even documented cases of people losing their lives because of problems with software!  

So while it's fun to find bugs, it's also critically important.  And it's even more important to remember that the true test of software is how it behaves in production with real users.  Often testers keep their focus on their test environment, because that's where they have the most control over the software under test, but it's crucial to test in production as well.  

I have seen situations where testers only tested a new feature in their test environment, and then were totally surprised when users reported that the feature didn't work at all in production!  This was because there were environment variables that were hard-coded to match the test environment.  The feature was released to production, and the testers didn't bother to check it.

Having things "work" in production is only one facet of quality, however.  We also need to make sure that pages load within a reasonable amount of time, that data is saved correctly, and that the system behaves well under times of high use.

Take a moment to think about the application you test.  In production:

• Is it usable?
• Is it reliable?
• Is the user's data secure?
• Do the pages load quickly?
• Are API response times quick?
• Do you monitor production use, and are you alerted automatically if there's a problem?
• Can you search your application's logs for errors?

Saying "But it worked in the test environment" is the tester's equivalent of the developer saying "But it worked on my machine".  It's fun to test and find bugs.  It's fun to check items off in test plans.  It's fun to see test automation run and pass. But none of those things matter if your end user has a poor experience with your application.  

Adventures in Node: Promises

Have you ever written an automated UI test that uses Javascript, and when you went to assert on a response, you got Promise {pending} instead of what you were expecting?  This really frustrated me when I first encountered it!  A developer I was working with explained that this is because Javascript processes commands asynchronously through the use of promises.  I sort of understood what he meant, so I tried to work with it as best I could, but I didn't really get it.

As I mentioned in this post, I've been taking a really awesome course on Node.js.  It's much more extensive than any programming language course I've ever taken, even the ones I took in college.  So I'm starting to understand Node concepts more clearly, and one of those concepts is promises!  In this post I'll explain why Javascript and Node need promises and show an example of how they work.

Javascript needs promises because it is a single-threaded language, meaning it can only do one thing at a time.  If we had a program where we needed to do three things, such as make an http request, alphabetize a list, and update a record in a database, we wouldn't want to have to wait around for each of those tasks to finish before we went on to the next one, because our program would be very slow!  So Javascript is designed to be asynchronous- it can start a task, and then while it's waiting for that task to complete, it can start the next task.

Our program with three things might actually run like this:
start the http request
start alphabetizing the list
start updating the record in the database
finish alphabetizing the list
finish updating the record in the database
finish the http request

The way that Javascript and Node manage this is through the use of promises.  Let's take a look at a promise:

const sumChecker = new Promise((resolve, reject) => {
     if (a+b==c) {
          resolve('You are correct!)
     }
     else {
          reject('Sorry, your math is wrong.')
     }
}

This function called sumChecker is a promise.  It's going to have two possible results: resolve and reject.  If the sum is correct, it resolves the promise, and if it's incorrect, it rejects it.  All promises behave in this way; there will be an option to resolve the promise and an option to reject the promise.

When the promise is called, either resolve or reject will be returned; you can't ever return both.  Let's look at an example of calling the promise:

sumChecker.then((result) => {
     console.log('Success!', result)
}).catch((error) => {
     console.log('Error!', error)
})

The result that is returned from calling the promise will either be resolve or reject.  If the result is resolve, then the program knows to continue and will return the resolve message.  If the result is reject, then the program knows to throw an error and will return the reject message.

You can try this for yourself if you have Node installed!  Simply copy the promise and the call and paste them into your favorite code editor.  Then at the beginning of the file, add these lines:

var a = 1
var b = 2
var c = 3

Save the file with the name myfile.js, navigate in the command line to the file's location, and run the file with the command node myfile.js.  You should see this response: Success! You are correct!

If you make a change to the c variable and set it to 4, save and run the command again, you'll see this response: Error! Sorry, your math is wrong.

Let's put a log statement in between the promise and the call to the promise that looks like this: console.log(sumChecker), so we can see the state of sumChecker before we've called it. If we change the value of c back to 3 so we'll get a positive result, save the file, and run the program with node myfile.js now, we'll get the result Promise { 'You are correct!'} in addition to the response we got earlier.  That seems easy!  But the reason why we can get the promise resolved so quickly is because the sumChecker promise executes really fast.  Let's see what happens if we make the sumChecker work more slowly, like a real promise would.

Update the sumChecker promise to look like this:

const sumChecker = new Promise((resolve, reject) => {
     if (a+b==c) {
          setTimeout(() => {
               resolve('You are correct!)
          }, 2000)
     }
     else {
          reject('Sorry, your math is wrong.')
     }
}

All we're doing here is adding a two-second timeout to the resolved promise.  Save the file, and run the program again with node myfile.js.  This time you'll first get the result Promise { <pending> }, and after two seconds, you'll get the result Success! You are correct!  

Now it should be clear why you get Promise { <pending> } when you are making Javascript or Node calls.  It's because the promise hasn't completed yet.  This is why we use the .then() command.  We wait for the response to the call to come back, then we do something with the response.  If we're writing a test, at that point we can assert on our result.

I hope you'll take the time to try running this file with Node, because there's nothing quite like doing hands-on work to generate understanding.  You can try changing the variables or any of the response messages to get a feel for how it's working.  Here's the final version of the file if you'd like to copy and paste it:

var a = 1
var b = 2
var c = 3

const sumChecker = new Promise((resolve, reject) => {
    if (a+b==c) {
        setTimeout(() => {
            resolve('You are correct!')
        }, 2000)
    }
    else {
        reject('Sorry, your math is wrong.')
    }
})

console.log(sumChecker)

sumChecker.then((result) => {
    console.log('Success!', result)
}).catch((error) => {
    console.log('Error!', error)
})

Enjoy your new-found understanding of promises!

Book Review: Perfect Software and Other Illusions About Testing

"Perfect Software and Other Illusions About Testing", by Gerald Weinberg, is the best book on testing I have ever read.  It is a must-read for anyone who works with software: CEOs, CTOs, scrum masters, team leads, developers, product owners, business analysts, and software testers.

Before I get into why this book is so great, I'll first acquaint you with the author.  Gerald "Jerry" Weinberg (1933-2018) was involved in the creation of software for over fifty years.  Early in his career he worked for NASA on Project Mercury, the project that created spacecraft that allowed a human to orbit the earth.  For decades he consulted with companies about building quality software, and over those years he gained a great deal of wisdom about software testing.  "Perfect Software", which was published in 2014, seems to me to be the culmination of his years of experience.

The book is divided into several chapters, each of which looks at a particular aspect of software testing. Many examples are given from Jerry's consulting experience, and each chapter closes with a summary and a list of common mistakes that companies make.  Rather than summarizing the lessons he imparts, I think it would be best to include Jerry's own words here.  Here are some of my favorite quotes from the book:

"Before you even begin to test, ask yourself: What questions do I have about this product's risks?  Will testing help answer these questions?"

"There are an infinite number of possible tests...Since we can't test everything, any set of real tests is some kind of sample- a portion, piece, or segment that is in some way representative of a whole set of possible tests."

"Knowing about the structure of the software you're testing can help you to identify special cases, subtle features, and important ranges to try- all of which can help narrow the inference gap between what the software can do and what it will do during actual use."

"Testing gathers information about a product; it does not fix things it finds that are wrong."

"If you're going to ignore information or go ahead with predetermined plans despite what the tests turn up, don't bother testing."

"If you blame messengers for bringing news you don't want to hear, you'll be rewarded by not hearing the news you should hear."

"Quality is a product of the entire development process.  Poor testing can lead to poor quality, but good testing won't lead to good quality unless all other parts of the process are in place and performed properly."

"Testing starts at project conception, or before. If you don't know this, you don't understand testing at all."

"Without a process that includes regular technical reviews, no project will rise above mediocrity, no matter how good its machine-testing process."

"No developer is good enough to consistently do it alone, and do it right."

"Data are meaningless until someone determines their meaning.  Different people give different meanings to the same data.  Gather data, then sit down and ponder at least three possible meanings."

"When someone says, 'The response should be very fast', what does that mean, exactly?  What meanings do 'should', 'very', and 'fast' give to the stated information?"

"Numbers can be useful, but only if they're validated by personal observation and set in context by a story about them."

"Garbage arranged in a spreadsheet is still garbage."

Jerry uses many great hypothetical scenarios to illustrate his points, and he also uses real-world examples from his years of consulting.  Here are some of my favorites:

• The tester who didn't log a bug he found because it wasn't in "his" component
• The manager who thought that the project was ready to ship because they ran 600,000 test cases and "nothing crashed the system"
• The team who thought their biggest problem was their bug-tracking system, because the system couldn't handle their 140,000 open bugs
• The team who took so long to triage bugs that couldn't make a decision on any of them, resulting in 129 undiscussed and unfixed bugs
• The tester who assumed that her new automated test tool was working correctly because all the tests displayed in green at the end
• The developer-tester team who were gaming the bug bounty system by having the developer add bugs to the code, the tester find the bugs quickly, and the developer fix them just as quickly, resulting in rewards for both
• The VP of Development who wanted a really big written test plan so he could have something big to slam down on a desk to "prove" that they had tested well

If you would like to think about what role testing plays in your software development project, what constitutes a good test, how to plan testing for a project, or how to interpret test data in order to make management decisions, then "Perfect Software" is the book for you.  I plan to re-read this book every year to make sure that I have fully retained all the lessons it offers.

Rarely Used HTTP Methods

A couple of months ago, one of the developers I work with asked me to test a bug fix he'd done.  In order to test it, I'd need to make an HTTP request with the OPTIONS method.  I'd never heard of the OPTIONS method, and it got me thinking: what other HTTP methods did I not know about?  In this post, I'll talk about four rarely used methods and and how you might use them in your testing.

OPTIONS:
This method returns whatever methods are allowed for a particular endpoint.  For example, if you had a URL called http://cats.com/cat, and you could use it to get a list of cats or add a cat, the methods that the OPTIONS request would return would be GET and POST.

OPTIONS demo:
Let's use the Restful-Booker API to try out the OPTIONS method.  Assuming you have Postman installed, we'll create a new GET request that calls this URL: https://restful-booker.herokuapp.com/booking. When you run this request, you'll see a list of all the available hotel-bookings for the app in the response body. Now let's change the method from GET to OPTIONS. When you run this request, you'll see GET, HEAD, and POST in the response body. These are the three methods that are available for this endpoint.

Why would you use the OPTIONS method?
If you are testing an API, this is a great way to find out if there are any valid endpoints that you don't know about. This can reveal more features for you to test, or it could potentially reveal a security hole. For example, maybe your API shouldn't really have a DELETE method, but someone implemented it by mistake.

HEAD:
This method returns only the headers of the response to a GET request. It's used if you want to check the response headers without putting pressure on the server to return other data.

HEAD demo:
We'll use the very same URL that we used for our OPTIONS demo. First, let's return our method to GET. Run the request, and see that you get a response body with the list of available bookings. Take a look at the headers that were returned with the response: Server, Connection, X-Powered-By, Content-Type, Content-Length, Etag, Date, and Via. Now let's change our verb to HEAD and re-run the request. We won't get anything in the body of the response, but we will get those same eight headers.

Why would you use the HEAD method?
This method would be a great way to check the headers of a GET response without having to actually return data. Headers are important because they often help to enforce security rules. If you know what headers your API should be returning, you can run this request on all of your endpoints to make sure that the right headers are being used.

CONNECT:
This method establishes a tunnel to the server that is identified by a URL. It's often used for proxy connections.

CONNECT demo:
For this demo, you'll need to have curl enabled. You can check to see if curl is installed on your machine by typing curl --version in your command line window. If you get a version back, you have curl installed.

To try out CONNECT, type this command into your command line window: curl -v CONNECT http:///kristinjackvony.com. Take a look at the response you get; about nine lines from the bottom, you'll see the message "301 Moved Permanently". This is because I recently changed this domain name to point to my Thinking Tester webpage instead of my personal webpage. I didn't do that because of this tutorial, but it wound up being useful!

Why would you use CONNECT?
You'd use CONNECT any time you want to see exactly what happens when you try connecting to an HTTP resource. This could be helpful with security testing, and any time you are using a proxy.

TRACE:
This method is similar to CONNECT in that it connects to a resource, but it also tries to get a response back.

TRACE demo:
We'll use curl again to try out TRACE. Type this command in the command window: curl -v TRACE http://isithalloween.com. You'll get back some response headers, plus the source code for the page.

Why would you use TRACE?
This would be good for security testing. Because you get the source code for the page in the response, you can inspect it to see if there are any cookies or authentication headers that a malicious user could exploit.

I hope you've gotten some good testing ideas from these rarely used HTTP methods! In my research I found all kinds of other methods that appear to no longer be in use, such as COPY, LINK, UNLINK, LOCK, and UNLOCK. Have you ever used these, or other rare methods? Tell me about it in the comments!