How to Design a Load Test

Last week, we talked about Performance Testing and various ways to measure the reliability and speed of your application.  This week we'll be talking about Load Testing.  Load testing is simply measuring how your application will perform under times of great demand.  This could mean testing scenarios of reasonable load, or it could mean testing with scenarios of extreme stress to find the limits of the application.

It's easy to find a load testing tool, create a few tests, and run them with a few hundred users to create metrics.  But this isn't particularly helpful unless you know why you are testing and how your results will help you.

So before you begin any load testing, it's important to ask the following questions:

• What kind of scenarios are you testing for?
• What is the expected behavior in those scenarios?

Let's imagine that you have a new website that sells boxes of chocolates.  You have noticed that your site is most active on Saturday mornings.  Also, Valentine's Day is coming, and you anticipate that you will have many more orders in the week leading up to that day.  In March, your company will be reviewed by a popular cable TV show, and you are hopeful this will lead to thousands of visits to your site.

With this in mind, you come up with the following expected behaviors:

• You would like your web pages to load in less than two seconds under a typical Saturday morning load
• You would like to be able to process five hundred orders an hour, which will get you through the busy Valentine's Day season
• You would like your site to be able to handle ten thousand visitors in an hour, which is how many people you expect to visit right after the TV show airs

The next step is to figure out what test environment you will use.  Testing in Production would obviously provide the most realistic environment, but load testing there might be a bad idea if your testing results in crashing your site!  The next most realistic option would be a test environment that accurately mimics your Prod environment in terms of the number of servers used and the size of the back-end database.  An ideal situation would be one in which this test environment was only used for your load testing, but this is not always an option; you may need to share this environment with other testers, in which case you'll need to be aware of what kinds of tests they are running and how they will impact you.  You'll also want to let other testers know when you are conducting your load tests, so they won't be surprised if response times increase.  

Once your test environment is ready, you can conduct some simple baseline tests.  You can use some of the strategies mentioned in last week's post to find out how long it takes for your web pages to load, and what the typical response times are for your API endpoints.  Knowing these values will help you gauge how large an impact a load scenario will have on your application.

Now it's time to design your tests.  There are a couple of different strategies to use in this process:

• You can test individual components, such as loading a webpage or making a single request to an API
• You can test whole user journeys, such as browsing, adding an item to a cart, and making a purchase

You'll probably want to use both of these strategies, but not at the same time.  For instance, you could measure how long it takes to load the home page of your website when you have ten thousand requests for the page in an hour.  In a separate test, you could create a scenario where hundreds of users are browsing, adding items to their cart, and making a purchase, and you could monitor the results for any errors.  

For each test you design, you'll want to determine the following:

• How many users will be interacting with the application at one time?
• Will those users be added all at once, or every few seconds?
• Will the users execute just one action and then stop, or will they execute the action continuously for the duration of the test?
• How long will the test last?

Let's design a test for the Valentine's Day scenario.  We'll assume that you have created test steps that will load the webpage, browse through three product pages, add one product to the cart, and make a purchase.  We already mentioned above that you'll want to be able to handle five hundred orders an hour, so we'll set up the test to do just that.  It's unlikely that in a real-world scenario all five hundred users would start the ordering process at the same time, so we'll set the test to add a new user every five seconds.  Each user will run through their scenario once and then stop.  The test will run for one hour, or until all five hundred users have completed the scenario.

Before you run your test, be sure that your test steps will return errors if they don't result in the expected response.  When I first got started with load testing, I ran several tests with hundreds of requests before I discovered that all of my requests were returning an empty set.  Because the requests were returning a 200 response, I didn't notice that there was anything wrong!  Make sure that your steps have assertions that will validate that your application is really behaving as you want it to.

Once you have the test steps in place, you've made sure that the steps have good assertions, and you have your test parameters set up with the number of users, the ramp-up time (how frequently a new user will be added to the test), and the test duration, it's time to run the test!  While the test is running, watch the response times and the CPU usage of your application.  If you start seeing errors or high CPU spikes, you can stop the test and make note of how high the load was when the spikes occurred.

Whether you need to stop the test early or whether the test completed successfully, you'll want to run a few test passes to make sure that your data is fairly consistent.  At the end of your testing, you'll be able to answer the question: can my website handle 500 orders in an hour?  If all of the purchases were completed with no errors, and if all of the response times were reasonable, then the answer is yes.  If you started seeing errors, or if the response times increased to several seconds, then the answer is no. If the answer is no, you can take the data you collected and share it with your developers, showing them exactly how many users it took to slow the system down.

Load testing should never be an activity you do to simply check it off of a list of test types.  When you take the time to consider what behaviors you want to measure, how you want your application to behave, what scenarios you can run to test those behaviors, and how you can analyze your results, you will ensure that load testing is a productive and informative activity.