Automated Form Testing

Now that we have looked at all the different ways we can manually test forms, it's time to think about automating those tests so you don't have to run through all the tests again every time you need to do a regression test!  But before you jump into automation, think for a while about what tests you really want to run.  Let's take last week's form as our example again.

What sorts of things would we want to make sure still work when we do our regression test?

• We'll want to make sure that every field can be populated with data and saved
• We'll want to make sure that all of the required fields are still required, and that we get an appropriate error message when we leave a required field blank
• We'll want to make sure that validation rules are respected in each field
• We'll want to verify that both the Save and the Cancel buttons work correctly

As I've discussed in a previous post, automated tests involving the GUI can often be slow and flaky.  So we'll want to limit the number of times we spin up a browser for testing.  But we'd also like our tests to only assert on one thing at a time.  Here are my suggestions for a suite of tests that could be run on this form.  I've written them in Cucumber for ease of reading:

Scenario: Happy Path

Given I am adding a new user

When I fill out all the fields and click the Save button

And I navigate to the Users page
Then I verify that all of the fields are displayed on the page

Scenario: Required Fields

Given I am adding a new user

When I leave all fields empty

And I click the Save button

Then I verify that an error message is present for each required field

Scenario: Validation Rules

Given I am adding a new user

When I give each field an invalid value

And I click the Save button

Then I verify that an error message is present for each invalid field

Scenario: Cancel Button

Given I am adding a new user

When I give each field a value

And I click the Cancel button

Then I verify that all fields are now empty

So, we can cover a lot of territory with just four automated tests!  

Our Happy Path tests that values in all of the fields will save correctly, and it also tests the Save button.  If one of the values does not save, we will know we have an error.  If all of the values do not save, we will know there is a problem either with the Save button or with the underlying data store. 

Our Required Fields test checks to make sure that every field that is required displays an appropriate error message if it is not filled out.  If even one required field does not display the message, we will know there is a problem.

Our Validation Rules test violates one rule for each text field.  It would be a good idea to mix and match the different types of rule violations.  For example:

First Name- send a value with numbers and non-letter characters, such as $23.00

Last Name- send a value with not enough letters, such as A

Personal Phone- send a value with the wrong number of digits, such as 008675309

Work Phone- send a value with non-number characters, such as 800*555-!@#$

Street Address 1- send a value with way too many characters, such as all the lyrics to "Frosty the Snowman" (my favorite song to test with)

Street Address 2- send a value with one too many characters

City- try sending a script, such as <script>alert("XSS Here!")</script>

State- send an invalid state, such as XX

Zip Code- send a Zip Code with the wrong number of digits, such as 0377

While this does not test every single way that the rules could be violated in every text field, it covers a wide variety of possibilities, and verifies that for each error, you receive an appropriate error message.  You could always add more tests here if you felt that more coverage was needed.

Finally, we verify that the Cancel button works correctly.  

With just these four automated tests, we're able to make sure no functionality is broken.  Running these regression tests will free you up to do more exploratory testing in your application, and to test new and more interesting features!

Testing Forms

Today is the day that we'll be putting it all together!  In the past few posts, we've been looking at different types of text fields and buttons; now we'll discuss testing a form as a whole.

There are as many different ways to test forms as there are text field types!  And unfortunately, testing forms is not particularly exciting.  Because of this, it's helpful to have a systematic way to test forms that will get you through it quickly, while making sure to test all the critical functionality.

Let's take a look at the form below, and I will walk you through a systematic approach to testing it.  For simplicity, we'll assume that the users will all be US-based.  (See my previous posts for discussions on testing international postal codes and phone numbers.)

Step One:  Required Fields

The first thing I do when I test a form is to make note of which fields are required.  This particular form denotes required fields with a red asterisk.  We can see that First Name, Last Name, Personal Phone, Street Address 1, City, State, and Zip Code are all required.  I want to verify that the form cannot be submitted when a required field is missing and that I'm notified of which field is missing when I attempt to submit the form.
1. I click the Save button when no fields have been filled out. I make sure that all of the required fields have error messages.
2. I fill out the two non-required fields, click the Save button, and verify that all the required fields have error messages.
3. I fill out one required field and click the Save button, and I verify that all the other fields have error messages.  I start with submitting just the First Name, then just the Last Name, then just the Personal Phone, etc., until I have cycled through all of the required fields.
4. I try various combinations of two, three, four, and five required fields to make sure the appropriate error messages are displayed.  I don't concern myself with testing every possible combination, since it's highly likely that the required field will behave similarly in these situations.
5. I try filling out all the required fields but one, and verify that I still receive an error for that field.  I cycle through all the possibilities of having one missing required field.
6. I fill out all of the fields- required and non-required- except for one required field, and verify that I still receive an error.
7. I fill out all of required fields and click the Save button, and I verify that no error messages appear and that the entered fields have been saved correctly to the database.
8. Finally, I fill out all of the fields- required and non-required, and I verify that no error messages appear and that the entered fields have been saved correctly to the database.

Step Two: Field Validation

The next thing I do is verify that each individual text box has appropriate validation on it.  First, for each text field, I discover what the upper and lower character limits are.  I enter in all the required fields except for the one text box I am testing.  Then in that text box:
1. I try entering just one character
2. I try entering the lower limit of characters, minus one character
3. I try entering the upper limit of characters, plus one character
4. I try entering a number of characters far beyond the upper limit
In all of these instances, the form should not save, and I should receive an appropriate error message.
Next:
5. I enter the lower limit of characters, and verify that the form is saved correctly
6. I enter the upper limit of characters, and verify that the form is saved correctly

Now that I have confirmed that the limits on characters are respected, it's time to try various letters, numbers and symbols.  For each text field, I find out what kinds of letters, numbers, or symbols are allowed.  For example, the First and Last Name fields should allow apostrophes (ex. O'Connor) and hyphens (ex. Smith-Clark), but should probably allow no numbers or symbols.  For each text field:
1. I try entering all of the allowed letters, numbers and symbols
2. I try entering the letters, numbers, or symbols that are not allowed, one at a time, until I have verified that they are all not allowed

For fields that have very specific accepted formats, I test those formats now.  For instance, I should be able to enter a Zip Code of 03773-2817, but not one of 03773-28.  For the State field, the two-letter code I put into the field should be a valid state, so I should be able to enter MA, but not XX.

Even though I've already tested all the forbidden numbers and symbols, I try a few cross-site scripting and SQL injection examples, to make sure no malicious code gets through.  (More on this in a future post.)

Step Three: Buttons

Even though I have already used the Save button dozens of times at this point, there are still a few things left to test here.  And I have not yet used the Cancel button.  So:
1. I click the Save button several times at once, and verify that only one instance of the data is saved
2. I click the Save button and then the Cancel button very quickly, and verify that the data is saved and that there are no errors
3. I click the Cancel button when no data has been entered, and verify that there are no errors
4. I click the Cancel button when data has been entered, and verify that the data is cleared.  I'll try this several times, with various combinations of required and non-required fields.
5. I click the Cancel button several times quickly, and verify that there are no errors

Step Four: Data

Finally, it's time to look at the data that is saved.  In earlier steps, I've already done some quick checks to determine if the data that I am entering is saved to the database.  Now I'll check:
1. That bad data I've entered is not saved to the database
2. That good data I've entered is saved to the database correctly
3. That good data I've entered is retrieved from the database correctly, and displayed correctly on the form

If the form I am testing has Edit and Delete functionality, it's important to test these as well, but I've covered these in my post CRUD Testing Part II- Update and Delete.

Once I've gone through a form in this systematic way, it's almost certain I will have found a few bugs to keep the developers busy.  It's tedious, yes, but once the bugs have been fixed I can move on to automation testing with confidence, knowing that I have really put this form through its paces!

Testing Buttons

Buttons tend to be something that it's easy to forget about.  The "Save" button is so ubiquitous that it seems like it would just work.  But overlooking testing buttons on a page can also mean overlooking bugs.  Recently someone told me about new functionality she was testing on an existing web page.  The new feature worked great, but her team forgot to test the "Delete" button.  It turned out that the developers had forgotten to account for the delete action in their new feature, and now "Delete" did nothing! 

Here are a few things to think about when testing buttons:

1. Test the "happy path" of the button.  Usually buttons have some sort of message on them that tell you what they are supposed to do.  So try it out, and make sure that the button delivers on its promise!  Did the "Save" button really save your data?  Did the "Delete" button delete it?  Did the "Clear" button clear it?  Did the "Search" button execute a search?  (Note: a "Back" button is also common, but back buttons are tricky enough that I will be saving them for a separate post.)

2. Misuse the button.  For example, quickly press the "Save" button twice when adding data.  Were two records saved instead of one?  Does a button get confused when you press it twice?  What about when you quickly press one button and then another?  One of the most amusing bugs I have found in my career was a "Refresh" button that redrew the screen, making the button larger whenever I pressed it.

3. Is the button there when it's supposed to be?  When we have spent a lot of time testing an application, it's easy to get used to the page and not notice when things are missing.  Think about your user stories when looking at the page.  What would your user need when doing certain activities on that page?  Follow the path that the user would take, and check for the buttons as you go.

4. Think about when the button is enabled and when it is disabled.  Does it make sense?  For example, is the "Save" button only enabled when all the required fields on your form have been filled out?  Is the "Clear" button only enabled when a field has been dirtied?  How do you know the button is enabled?  Can you tell by looking at it?  Does the button look enabled when it isn't?  Does the button look disabled when it's really active?  What happens when you press the button and it's not enabled?  Do the enabled and disabled rules for the button make sense?

5. Finally, see if you can hack your buttons!  For example, if you have a "Save" button that is disabled on your form because some of the required fields are missing, can you edit the html on the page so that it is enabled, and can you then use the button?  If you view the html and you see that there is a hidden button on the page, can you make it visible and active?  At best, that bug is an annoyance that will get inaccurate data into the database.  At worst, the bug represents a way that a hacker can infiltrate your system!  Imagine that you have a button that should only be visible and enabled if a user is an administrator.  If a malicious user can make that button appear on the screen and be active, they will have access to pages or features that only an admin should have. Your developer should include checks whenever a button is clicked on to make sure that the user has the rights to do what the button does. 

If you have never edited the html on a web page while testing, here are a few simple instructions for the Chrome browser:
a. Click on the three-button menu on the top right of your screen, and choose "More Tools -> Developer Tools".  A new panel will appear on the bottom or the right side of your screen.
b. Right-click on the button you want to test, and click "Inspect".  In the developer tools panel, you will now see the html for that button highlighted.
c. Right-click on that highlighted text and choose "Edit as HTML".  An editable text window will open up. 
d. If you see text such as "'disabled'=disabled" delete the text.  Click away from the editable field, and see if your button is now enabled. If it is, click on it and see what happens!
e. To find hidden buttons, look at the html in the developer tools, and use the search bar to search for "button".
f. If you find a button with the markup "ng-hide", try changing it to "ng-show".  See if you can get the button to appear on the page!

Buttons are one of the most important things to test.  Just imagine a user's frustration if the button they are trying to use is disabled, or doesn't do what they expect it to.  By being diligent in our testing, we can ensure that our users will want to work with our application. 

Testing Phone Fields- Part II- International Phone Numbers

If I were to be given three wishes for the world, my third wish (after world peace and ending world hunger) would be to standardize international phone numbers, because testing them is so complicated!  Let's take a look at some phone number patterns from around the world:

Mexico:
Phone numbers are ten digits, plus an area code of either two digits or three digits.  So the local number will be either twelve or thirteen digits.

Italy:
Landline numbers are generally nine to eleven digits, but some can be as short as six.  Mobile numbers are usually ten digits, but there are also some nine-digit numbers.

Japan:
Telephone numbers in Japan have an area code, an exchange number, and a subscriber number.  Area codes can have between two and five digits.  Generally the length of the entire number is limited to nine digits, so if the area code is longer, the exchange and subscriber numbers will be shorter.

Now let's think about international calling codes.  Calling codes can range from one digit (such as the US's +1) to three digits (such as +351 for Portugal).  So between the international calling codes and the phone number itself, there is a wide range of the number of digits that might be expected for a phone number.

Finally, let's think about number separation.  While US numbers are always separated in a 3-3-4 pattern, in other countries the numbers can be grouped in various ways.  Here are just some of the ways that phone numbers are grouped in England:

(xxx) xxxx xxxx
(xxxx) xxx xxxx
(xxxxx) xxxxxx
(xxxx xx) xxxxx

Therefore, it is extremely difficult to validate on whether the pattern of numbers, parentheses, and dashes entered by a user is correct.

So how on earth are we to validate international phone numbers?  The most important thing to do is to make sure that the developer doesn't try to come up with a validation regex on their own- that will make them (and you) go crazy!  Fortunately an international phone number formatting standard called E.164 has been developed.  Many companies such as Microsoft and Google have come up with regex patterns that can be used for E.164 validation.  For testing, there are free websites that will let you know if an international phone number is valid.  Most of these sites will require some sort of registration.  You can come up with a list of international numbers to test, verify with the website to see if they are valid, and then try them in your application.  For negative testing, simply use a sampling of invalid numbers and verify that you get an appropriate error message.

What format should a user use when entering in their phone number?  Some users might want to put parentheses, spaces, or dashes into their number.  Then there is the plus sign, which is used to indicate a country code.  For simplicity's sake, it would be best to expect nothing but digits and a plus sign.  It would also be good to prompt the user for what format the application is expecting.  For example, in the United States you might have a message such as: "Enter your phone number with no dashes, parentheses, or spaces.  For numbers outside the US, a plus sign (+) and the country code should be used."  Then for validation, if the number does not have a plus sign, simple US number validation rules can be applied, and if the number does have a plus sign, international validation rules can be applied.

Similarly, you may want to store the number in the database with the plus sign, to indicate that the number is an international one.  How should the numbers be displayed when retrieved in the application?  For many countries it's nearly impossible to know how the numbers should be spaced, unless the validation code you are using returns that to you.  If your country has a standard number formation, you can display the number in that standard.  If your country does not have a standard formation, or if you are displaying a number from outside your country, you can simply display the number as is, such as: +442073238299.

When it comes to international phone numbers, there really are no easy answers, which is why I am wishing for world standardization!  There are other ways to (mostly) solve the problem, such as 1) expecting only between 6 and 20 digits, not validating the number beyond making sure that it's within that number of digits, and displaying the number as entered, or 2) having a separate field for the country code, validating the number for that specific country, and formatting the number according to that validation.  What is most important is that you communicate closely with the developer to decide on a clear strategy and test it accordingly.