Email Subscription Form

Saturday, June 15, 2019

Why The Manual vs. Automation Debate is Wrong

I don't generally editorialize in my blog- I prefer to focus on what to test rather than theories of testing- but I feel compelled to say that I'm tired of the whole "manual vs. automated testing" discussion.  Some people describe automated testing as the cure for bad code everywhere, and others lament the poor manual tester who has no technical skills.  Meanwhile, there are advocates who say that automation is merely a panacea, and that automation code should be used only for simple tools that will aid the manual tester, who is the one who really knows the product.


In my opinion, this debate is unnecessary for two reasons:

1) "Manual" and "automated" are arbitrary designations that don't really mean anything.  If I write a Python script that will generate some test data for me, am I now an automation engineer?  If I log into an application and click around for a while before I write a Selenium test, am I now a manual tester?  

2) The whole point of software testing- to put it bluntly- is to do as much as we can to ensure that our software doesn't suck.  We often have limited time in which to do this.  So we should use whatever strategies we have available to test as thoroughly as we can, as quickly as possible.  

Let's take a look at three software testers: Marcia, Cindy, and Jan.  Each of them is asked to test the Superball Sorter (a hypothetical feature I created, described in this post).  

Marcia is very proud of her role as a "Software Developer in Test".  When she's asked to test the Superball Sorter, she thinks it would be really great to create a tool that would randomly generate sorting rules for each child.  She spends a couple of days working on this, then writes a Selenium test that will set those generated rules, run the sorter, and verify that the balls were sorted as expected.  Then she sets her test to run nightly and with every build.  

Unfortunately, Marcia didn't take much time to read the acceptance criteria, and she didn't do any exploratory testing.  She completely missed the fact that it's possible to have an invalid set of rules, so there are times when her randomly generated rules are invalid.  When this happens, the sorter returns an error, and because she didn't account for this, her Selenium test fails.  Moreover, it takes a long time for the test to run because the rules need to be set with each test and she needed to build in many explicit waits for the browser to respond to her requests.  

Cindy is often referred to as a "Manual Tester".  She doesn't have any interest in learning to code, but she's careful to read the acceptance criteria for the Superball Sorter feature, and she asks good questions of the developers.  She creates a huge test plan that accounts for many different variations of the sorting rules, and she comes up with a number of edge cases to test.  As a result, she finds a couple of bugs, which the developers then fix.

After she does her initial testing, she creates a regression test plan, which she faithfully executes at every software release.  Unfortunately, the test plan takes an hour to run, and combined with the other features that she is manually testing, it now takes her three hours to run a full regression suite.  When the team releases software, they are often held up by the time it takes for her to run these tests.  Moreover, there's no way she can run these tests whenever the developers do a build, so they are often introducing bugs that don't get caught until a few days later.

Jan is a software tester who doesn't concern herself with what label she has.  She pays attention during feature meetings to understand how the Superball Sorter will work long before it's ready for testing.  Like Cindy, she creates a huge test plan with lots of permutations of sorting rules.  But she also familiarizes herself with the API call that's used to set the sorting rules, and she starts setting up a collection of requests that will allow her to create rules quickly.  With this collection, she's able to run through all her manual test cases in record time, and she finds a couple of bugs along the way.

She also learns about the API call that triggers the sorting process, and the call that returns data about what balls each child has after sorting.  With these three API calls and the use of environment variables, she's able to set up a collection of requests that sets the rules, triggers the sorting, and verifies that the children receive the correct balls.  

She now combines features from her two collections to create test suites for build testing, nightly regression testing, and deployment testing.  She sets up scripts that will trigger the tests through her company's CI tool.  Finally, she writes a couple of UI tests with Selenium that will verify that the Sorter's page elements appear in the browser correctly, and sets those to run nightly and with every deployment.

With Jan's work, the developers are able to discover quickly if they've made any changes in logic that cause the Superball Sorter to behave differently.  With each deployment, Jan can rest assured that the feature is working correctly as long as her API and UI tests are passing.  This frees up Jan to do exploratory testing on the next feature.

Which of these testers came up with a process that more efficiently tested the quality of the software?  Which one is more likely to catch any bugs that come up in the future?  My money's on Jan!  Jan isn't simply a "manual tester", but she isn't a "software developer in test" either.  Jan spends time learning about the features her team is writing, and about the best tools for testing them.  She doesn't code for coding's sake, but she doesn't shy away from code either.  The tools and skills she utilizes are a means to ensure the highest quality product for her team.  

32 comments:

  1. In your example for Jan, the 2 activities she does, the testing part and the automation have almost nothing to do with each other. The purpose of both are very different.

    In addition, the reality in the industry is that very few people know how to do 'manual' testing. The industry is dominated by test automation. It bites that automation proponents console 'manual' testers with, 'automation helps you explore'. You can prove me wrong by showing me an example of a test automation expert demonstrating a credible example of 'manual' testing.



    ReplyDelete
    Replies
    1. Hi Nilan- I agree that the manual/exploratory testing that Jan does and the automation that she sets up are different activities. But both of them have the same purpose in that they serve the end goal of making sure the software isn't buggy.

      I work with people who are good automation engineers and good manual testers. Perhaps it doesn't look like there are people who are good at both in social media, because bloggers and conference speakers often have a specific focus.

      Delete
  2. I just wonder, when Jan has the time to do all this.
    Or "Jan" is the team name, that consist of Cindy and Marcia, where Cindy knows the product, and helps Marcia with test cases, while Marcia spends her time converting those test cases to API and UI automated tests?

    ReplyDelete
    Replies
    1. Hi Dmitry- Jan is her own person in this example! I do all the activities that Jan does in my position (with the exception of UI automation, which my colleague is doing), and I don't have trouble finding the time to do them. It is possible to be very familiar with your product while at the same time contributing to test automation.

      Delete
  3. Good story. Easy to choose which tester we should follow. Thanks for the article.

    ReplyDelete
  4. Hi Dear

    Thanks for sharing the Test Automation Framework blog with us. It is a very useful post for me and others. I'm waiting for your next post.

    ReplyDelete
  5. This comment has been removed by a blog administrator.

    ReplyDelete
  6. Thanks for sharing nice post and nice urging commented at this place, I am in fact enjoying by these.I like visiting your site since I always come across interesting articles like this one. Keep sharing! Regards. Read more about
    testing services companies
    Software Testing and Quality Assurance Services
    Performance testing services
    Security testing services
    software testing company

    ReplyDelete
  7. Thanks for sharing such useful information.
    Manual testing

    ReplyDelete
  8. Very useful post and I think it is rather easy to see from the other comments as well that this post is well written and useful. Keep up the good work
    software testing services
    software testing companies
    Security testing services

    ReplyDelete
  9. Good details and really helpful information about manual and automation testing

    ReplyDelete
  10. Thanks for Sharing.
    It's great to be here and to learn more about software testing.
    Software testing training and placement in chennai

    ReplyDelete
  11. software testing company in India
    software testing company in Hyderabad
    Thanks for sharing such a useful information about Manual vs. Automation.
    Informative blog.

    ReplyDelete
  12. software testing company in India
    software testing company in Hyderabad
    Thanks for sharing such a useful information about Manual vs. Automation.
    Informative blog.
    keep sharing.

    ReplyDelete
  13. Wow it is really wonderful and awesome thus it is veWow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your site.ry much useful for me to understand many concepts and helped me a lot.
    Manual Testing Training in Bangalore

    ReplyDelete
  14. I am doing automated testing training and so I am so fond of these kind of blogs to understand it better for my future. Thanks for sharing this blog. Great information.

    ReplyDelete
  15. Outsourcing Training institute In Dhaka Said,
    Very interesting, good job and thanks for sharing such a good blog.

    ReplyDelete
  16. Hello,I love reading through your blog, I wanted to leave a little comment to support you and wish you a good continuation. Wishing you the best of luck for all your blogging efforts. Web Designing Services in Hyderabad

    ReplyDelete
  17. Your blog has all the information I was looking for for a long time. I want to thank you for sharing such important information with us. I will share your blog with my friends. I wish you to keep sharing such important blogs with us in the future also. Apart from this, if you ever need Manual And Automation Testing, then you can get all the information by clicking on the given link.

    ReplyDelete
  18. I am really very happy to visit your blog. Directly I am found which I truly need. please visit our website for more information
    Quality Engineering Services in USA

    ReplyDelete

  19. thanks for the information..
    inovies

    Web Development Company in Hyderabad

    ReplyDelete
  20. Timber dining Chairs

    Timber dining chairs set are the finest & spectacular wooden dining chairs. Timber dining chairs are durable & very comfy.

    Timber wood's biggest advantage is that it is strong and durable and lasts long as you can't afford to have a loosely carved chair for everyday dining use.

    ReplyDelete
  21. "Why The Manual vs. Automation Debate is Wrong" is a thought-provoking and insightful perspective that challenges conventional thinking. Autodesk Maya Better Than Blender This well-reasoned article encourages communication.

    ReplyDelete
  22. slot sultanking is a good place to play game and get a lot of benefit

    ReplyDelete
  23. It’s really a great and helpful piece of information. I am
    happy that you shared this useful information with us. Please stay us up to date like
    this. http://alternatifsultanking.com/

    ReplyDelete

New Blog Location!

I've moved!  I've really enjoyed using Blogger for my blog, but it didn't integrate with my website in the way I wanted.  So I&#...