Mobile Testing Part III: Seven Automated Mobile Testing Tips (and Five Great Tools)

Walk into any mobile carrier store and you will see a wide range of mobile devices for sale.  Of course you want to make sure that your application works well on all of those devices, in addition to the older devices that some users have.  But running even the simplest of manual tests on a phone or tablet takes time.  Multiply that time by the number of devices you want to support, and you've got a huge testing burden!

This is where automated mobile testing comes in.  We are fortunate to be testing at a time where there are a whole range of products and services to help us automate our mobile tests.  Later in this article, I will discuss five of them.  But first, let's take a look at seven tips to help you be successful with mobile automated testing.

Tip 1: Don't test things on mobile that could be more easily tested elsewhere

Mobile automation is not the place to test your back-end services.  It's also not the place to test the general logic of the application, unless your application is mobile only.  Mobile testing should be used for verifying that elements appear correctly on the device and function correctly when used.  For example, let's say you have a sign-up form in your application.  In your mobile testing, you'll want to make sure that the form renders correctly, that all fields can be filled in, that error messages display appropriately, and that the Save button submits the form when clicked on.  But you don't want to test that the error message has the correct text, or that the fields have all saved correctly.  You can save those tests for standard Web browser or API automation.

Tip 2: Decide whether you want to run your tests on real devices or emulators

The advantage of running your tests on real devices is that the devices will behave like the devices your users own, with the possibility of having a low battery, connectivity issues, or other applications running.  But because of this, it's more likely that your tests will fail because a phone in the device farm froze up or was being used by another tester.  Annoyances like these can be avoided by using emulators, but emulators can't completely mimic the real user experience.  It's up to you decide which choice is more appropriate for your application.  

Tip 3: Test only one thing at a time

Mobile tests can be flaky, due to the issues found in real devices discussed above and other issues such as the variations found in different phones and tablets.  You may find yourself spending a fair amount of time taking a look at your failed tests and diagnosing why they failed.  Because of this, it's a good strategy to keep your tests small.  For example, if you were testing a login screen, you could have one test for a successful login and a second test for an unsuccessful login, instead of putting both scenarios into the same test.

Tip 4: Be prepared to re-run tests

As mentioned in Tip 3, you will probably encounter some flakiness in your mobile tests.  A test can fail simply because the service that is hosting the emulator loses connectivity for a moment.  Because of this, you may want to set up a system where your tests run once and then re-run the failed tests automatically.  You can then set up an alert that will notify you only if a test has failed twice.

Tip 5: Don't feel like you have to test every device in existence

As testers, we love to be thorough.  We love to come up with every possible permutation in testing and run through them all.  But in the mobile space, this can quickly drive you crazy!  The more devices you are running your automated tests on, the more failures you will have.  The more failures you have, the more time you have to spend diagnosing those issues.  This is time taken away from new feature testing or exploratory testing.  Do some research on which devices your users own and come up with a list of devices to test with that covers most, but not all, of those devices.  

Tip 6: Take screenshots

Nothing is more frustrating than seeing that a test failed and not being able to figure out why.  Screenshots can help you determine if you were on the correct screen during a test step and if all the elements are visible.  Some mobile testing companies take a screenshot of every test step as the test progresses.  Others automatically take a screenshot of the last view before a test fails.  You can also code your test to take screenshots of specific test steps.  

Tip 7: Use visual validation

Visual validation is essential in mobile testing.  Many of the bugs you will encounter will be elements not rendering correctly on the screen.  You can test for the presence of an element, but unless you have a way to compare a screenshot with one you have on file, you won't really be verifying that your elements are visible to the user.  Applitools makes an excellent product for visual comparison.  It integrates with common test software such as Selenium, Appium, and Protractor.  With Applitools, you can build visual verification right into your tests and save a collection of screenshots from every device you test with to use for image comparison. 

Now let's discuss some good tools for test automation.  I've already mentioned Applitools; below are four other tools that are great for mobile test automation.  The mobile landscape is filled with products for automated testing, both open-source and paid.  In this post, I am discussing only the products that I have used; there are many other great products out there. 

Visual Studio App Center:  A Microsoft product that allows you to test Android and iOS applications on real devices.  A screenshot is taken of every test step, which makes it easy to figure out where a test started to go wrong. 

Appium:  An open-source product that integrates with Selenium and provides the capability to test on device emulators (or real devices if you integrate with a device farm). 

Sauce Labs:  Sauce Labs is great for testing on both mobile devices and web browsers on all kinds of operating systems.  You can run tests on real devices or emulators, and you can run tests in parallel.  They integrate well with Selenium and Appium.  A screenshot is taken whenever a test fails, and you can even watch a video of your test execution.

Perfecto: Uses real devices and integrates with Visual Studio, Appium, and Selenium.  They can simulate real-world user conditions such as network availability and location.

Whatever automated test tools you choose to use, remember the tips above, and you will ensure that you are comprehensively testing your application on mobile without spending a lot of time debugging. 

I initially said this series on Mobile Testing was going to be three blog posts long.  On reflection, I've realized that we need a fourth post: Mobile Security Testing.  This is a topic I know very little about.  So I'll be doing some research, and you can expect Mobile Testing Part IV from me next week!