Email Subscription Form

Saturday, October 26, 2019

The Power of Not Knowing

Recently I saw a tweet from Ben Simo (@QualityFrog) that mentioned that he sometimes likes to practice what he calls "intentional ignorance"- where he doesn't read some of the documentation or code for a new feature to see what he can find while doing exploratory testing.  His tweet reminded me that I used to do this too!

I haven't done this in a while, because the team I work on is a great Agile team.  The testers are invited to the feature grooming sessions, each story has acceptance criteria written, and the developers do a feature handoff with the testers when each story is ready for testing.

But at previous companies, I was often given a story to test with no feature handoff and no acceptance criteria.  Sometimes the story wouldn't even have a description, and would have some cryptic title, like "Endpoint for search".  I would usually be annoyed by this, and I would ask for clarification, but I would first use it as an opportunity to do some exploratory testing while I had no pre-conceived notions what the feature could do or not do.   And while testing in this fashion, I would often find a bug, show it to the developer, and have him or her say, "Oh, it never even occurred to me to test the feature in that way."


Of course I don't want to go back to the days of cryptic story titles with no description!  But testing without knowing what the feature does can have some benefits:

  • You approach the application the same way a user would.  When your users see your new feature for the first time, they don't have the benefit of instructions.  By trying out the feature without knowing how it works, you could discover that an action button is hard to find, or that it's difficult to know what to do first on a multi-part form.  
  • You might try entering data that no one was expecting.  For example, there could be a form field where the date was supposed to be entered with month and day only, but you enter in the month, day, and year, which breaks the form.  
  • Without any instructions from the developer, you might think of other features to test the new feature with, besides those the developer thought of.  Those feature combinations might yield new bugs.

So how can we add these advantages back into our testing without skipping reading the acceptance criteria and having feature handoffs?  Here are a few ways:

  • Pair test with someone on another team.  At my company we have many teams, each of which often has no idea what the other teams are building.  Four times a year, the software testers get together in pairs where the two testers are from very different teams, and they swap applications and start testing.  This is a great way to find bugs and user experience issues!
  • When you start testing, spend some time just playing around with the new feature before writing a test plan.  By exploring in this way, you might come up with some unusual testing ideas.
  • After you've tested the acceptance criteria, take some time to think about what features might be used with the new feature.  What happens when you test them together?  For example, if you were testing a new page of data, you could test it with the global sort feature that already exists in your application.

Of course, there are also times where not knowing all the details about a feature is detrimental.  There have been times in my testing career where I tested a feature and completely missed something that the feature could do, because no one told me about it.  That's why I'm glad that we have acceptance criteria and feature handoffs.  But there are also times when not knowing can yield some of the most interesting bugs.

5 comments:

  1. Nobody tells a user. I don't want to be told anything by a developer. They express themselves with the GUI they built.

    If you can't find the feature, neither will the user.

    ReplyDelete
    Replies
    1. That's true if you are a UX tester, David, but there are often things that QA engineers need to know about a feature in order for them to test completely, such as potential regression areas.

      Delete
  2. You said: "You might try entering data that no one was expecting. For example, there could be a form field where the date was supposed to be entered with month and day only, but you enter in the month, day, and year, which breaks the form."

    I was once out with a client who was a travel agency. Their booking app had a page for add-on costs, like transfers, excess luggage or surcharges. You put a monetary value in a field and that went straight through to the invoice.

    Except if you put a currency symbol in before the value, the app tripped over the special character and sent 0.00 through to the invoice. Now, the company had a very experienced and well-trained call centre team, so they probably wouldn't make that mistake. But if someone new came across this field, or if the additional cost was in a different currency, that symbol could get added to the input value. And the customer would be unlikely to complain about £0.00 appearing on their final invoice for something they expected to be charged for!

    ReplyDelete
  3. Week one with a new company saw me shut away in a small office with software and no documentation. The software had recently had a major overhaul and I was to give it a "quick check over". Just clicking on every button on every screen resulted in keeping the developers busy for weeks, noting field labels that didn't comply with UI standards added to that until it was suggested I stop testing for a while and go visit a few customers. The company hadn't had professional testers before...

    ReplyDelete
    Replies
    1. Wow, that's a great story, Chrissi! I had something similar happen at a company that had never had testers before, but it wasn't quite as extreme. Still, I found enough bugs to keep the devs busy for a year!

      Delete

Book Review: Agile Testing Condensed

I read a ton of books, and I've found that reading books about testing is my favorite way to learn new technical skills and testing stra...