Confused? Simplify!

As testers, we are often asked to test complex systems.  Gone are the days when testers were simply asked to fill out form fields and hit the Submit button; now we are testing data stores, cloud servers, messaging services, and much more.  When so many building blocks are used in our software, it can become easy to get overwhelmed and confused.  When this happens, it's best to simplify what we are testing until our situation becomes clear.

Here's an example that happened recently on my team: we were testing that push notifications of a specific type were working on an iPhone.  One of my teammates was triggering a push notification, but it wasn't appearing on the phone.  What could be wrong?  Maybe notifications were completely broken.  Maybe they were broken on the iPhone.  Maybe only this specific notification was broken.  Maybe only notifications of this type were broken.  In a situation where there are a lot of notifications to test and we are working on a deadline, this can become very confusing. 

So, we simplified by asking a series of questions and running a test for each one.  We started with:
Is this push notification working on an Android phone?
We triggered the same notification to go to an Android phone, and the push was delivered.  So we ruled out that the notification itself was broken.

Next, we asked:
Is this push notification working on any other iPhone?
We triggered the same notification to go to a different iPhone, and the push was delivered.  So we ruled out that the notification was broken on iOS devices.

Then we asked:
Is ANY notification working on this specific iPhone? 
We triggered some different notifications to go to the iPhone, and no pushes were delivered.  So we concluded that the problem was not with the notification, or with the push service; the problem was with the phone.

In taking a step back and asking three simple questions, we were able to quickly diagnose the problem.  Let's take a look at another example, using my hypothetical feature called the Superball Sorter, which sorts small and large colored balls among four children, as described in this post.

Let's imagine that we are testing a scenario where we are sorting the balls by both size and color.  We have the children set up with the following rules:
Amy gets only large balls
Bob gets only small purple balls and large red balls
Carol gets only small balls
Doug gets only green balls

When we run the sorter, a small purple ball is next in the sorting process, and it's Bob's turn to get a ball.  We are expecting that Bob is going to get the small purple ball because his sorting rules allow it, but he doesn't get the ball- it goes to Carol instead.  What could be wrong here?  Maybe Bob isn't getting any balls.  Maybe the purple ball isn't being sorted at all.  Maybe only the small balls aren't being sorted.  How can we figure out what is going on?

Our first question will be:
Can Bob get ANY sorted balls?  
We'll set up the sorter so Amy, Carol, and Doug only get large balls, and Bob only gets small balls.  We run the sorter, and Bob gets all the small balls.  So we know this isn't the problem.

Can anyone get the small purple ball?
Next, we'll set up the sorter so that Amy will only get small purple balls, and Bob, Carol, and Doug can get any ball at all.  We'll set up our list of balls so that the small purple ball is first on the list.  When we start our sorting process with Amy, she gets the small purple ball.  So now we know that the small purple ball isn't the problem.

Can Bob get the small purple ball in some other scenario?
We saw in our initial test that Bob wasn't getting the small purple ball, but can he EVER get that ball?  We'll set up our rules so that Amy will only get large balls, and Bob will get only small purple balls.  We won't give Carol and Doug any rules. Then we'll set up our list of balls so that the small purple ball is first on the list.  Amy won't get the small purple ball, because she only gets large balls, so the small purple ball is offered to Bob.  He gets the ball, so now we know that Bob can get the small purple ball in some scenarios.

At this point, we know that the problem is not the small purple ball.  What is different between the original scenario and the one we just ran?  One difference is that in the original scenario, all four children had a rule.  So let's ask this question:

Can Bob get the small purple ball when it's his only rule, and the other children all have rules?
We'll set up the rules like this:
Amy gets only large balls
Bob gets only small purple balls
Carol gets only small balls
Doug gets only green balls
We again set up our list of balls so that the small purple ball is first on the list.  The ball skips Amy, because it doesn't meet her rule, and Bob gets the ball.  So now we know that the problem is not that all the children have rules.  So now the next logical question is:

What happens when Bob has TWO rules?
We'll set up the rules like this:
Amy gets only large balls
Bob gets only small purple balls and small yellow balls
Carol gets only small balls
Doug gets only green balls

Our list of balls is the same, where the small purple ball is first.  This time, the ball skips Amy AND Bob, and Carol gets the small purple ball.

AHA!  Now we have a good working theory: when Bob has two rules, the sorting is not working correctly.  We can test out this theory by giving another child two rules, while giving everyone else one rule.  Are the balls sorted correctly?  What about when a child has two rules that specify color only and not size?  Will the two rules work then?  By continuing to ask questions, we can pinpoint precisely what the bug is.

By making your tests as simple as possible, you are able to narrow down the possibilities of where the bug is.  And by proceeding methodically and logically, you will be able to find that bug as quickly as possible, even in a very complex system.