Email Subscription Form

Saturday, October 12, 2019

Why You Should Be Testing in Production

This is a true story; I'm keeping the details vague to protect those involved.  Once there was a software team that was implementing new functionality.  They tested the new functionality in their QA environment, and it worked just fine.  So they scheduled a deployment: first to the Staging environment, then to Production.  They didn't have any automated tests for the new feature, because it was tricky to automate.  And they didn't bother to do any manual tests in Staging or Production, reasoning that if it worked in the QA environment, it must work everywhere.

You can probably guess what happened next- they started getting calls from customers that the new feature didn't work.  They investigated and found that this was true.  Then they tried out the feature in the Staging environment and found that it didn't work there either.  As it turned out, the team had used hard-coded configuration strings that were only valid in the QA environment.  If they had simply done ONE test in the Staging or Production environment, they would have noticed that something was wrong.  Instead, it was left to the customers to notice the problem.


There are two main reasons why things that work in a QA environment don't work in a Production environment:

1) Configuration problems- This is what happened with the team described above.  Software is complicated, and there are often multiple servers and databases that need to talk to each other in order for the software to work properly.  Keeping software secure means that each part of the application needs to be protected by passwords or other configuration strings.  If any one of those strings is incorrect, the software won't work completely.

2) Deployment problems- In this age of microservices, deploying software usually means deploying several different APIs.  In a large organization, there may be different teams responsible for different APIs.  For example, when a new feature in API A needs the new code in API B to work properly, API B will need to be deployed first.  It's possible that Team B will forget to deploy API B or not even realize that it needs to be deployed.  In cases like this, Team A might assume that API B had been deployed, and they will go ahead and deploy API A.  Without testing, Team A will have no way of knowing that the new feature isn't working.

By running tests in every environment, you can quickly discover if you have configuration or deployment problems.  It's often not necessary to go through extensive testing of a new feature in Production if you've already tested it in QA, but it is vital that you do at least SOME testing to verify that it's working!  We never want to have our customers find problems before we do.

10 comments:

  1. It's worth clarifying:
    - testing in production is in addition to testing that you would do?
    - testing in production is not a substitute for testing pre-release?

    Also worth thinking about architectural changes that aren't simple. In that case you can't just roll back once the feature is released.

    ReplyDelete
    Replies
    1. Hi Nilan- Yes! You should always do extensive testing in Pre-Prod environments. Testing in Prod is NOT a substitute for testing in QA. And whenever you go to Prod, you should have a rollback or mitigation strategy in place if something goes wrong. Thanks for these excellent points!

      Delete
    2. Hello all
      am looking few years that some guys comes into the market
      they called themselves hacker, carder or spammer they rip the
      peoples with different ways and it’s a badly impact to real hacker
      now situation is that peoples doesn’t believe that real hackers and carder scammer exists.
      Anyone want to make deal with me any type am available but first
      I‘ll show the proof that am real then make a deal like

      Available Services

      ..Wire Bank Transfer all over the world

      ..Western Union Transfer all over the world

      ..Credit Cards (USA, UK, AUS, CAN, NZ)

      ..School Grade upgrade / remove Records

      ..Spamming Tool

      ..keyloggers / rats

      ..Social Media recovery

      .. Teaching Hacking / spamming / carding (1/2 hours course)

      discount for re-seller

      Contact: 24/7

      fixitrogers@gmail.com

      Delete
  2. This comment has been removed by the author.

    ReplyDelete
  3. Hi Kristin, Thanks for the blog posts.
    I have question on this, so we tester's test and validate the release in test environment, but we are not involved after it has been released to production. So who will test the

    ReplyDelete
    Replies
    1. Hi Bharath- yes, you should be testing in pre-production and production! If this is not currently allowed at your company, share this article with your manager and see if you can get at least some access to every environment.

      Delete
  4. This comment has been removed by the author.

    ReplyDelete
  5. Hey Guys !

    USA Fresh & Verified SSN Leads with DL Number AVAILABLE with 99.9% connectivity
    All Leads have genuine & valid information

    **HEADERS IN LEADS**
    First Name | Last Name | SSN | Dob | DL Number | Address | City | State | Zip | Phone Number | Account Number | Bank Name | Employee Details | IP Address

    *Price for SSN lead $2
    *You can ask for sample before any deal
    *If anyone buy in bulk, we can negotiate
    *Sampling is just for serious buyers

    ==>ACTIVE, FRESH CC & CVV FULLZ AVAILABLE<==
    ->$5 PER EACH

    ->Hope for the long term deal
    ->Interested buyers will be welcome

    **Contact 24/7**
    Whatsapp > +923172721122
    Email > leads.sellers1212@gmail.com
    Telegram > @leadsupplier
    ICQ > 752822040

    ReplyDelete
  6. Hello all
    am looking few years that some guys comes into the market
    they called themselves hacker, carder or spammer they rip the
    peoples with different ways and it’s a badly impact to real hacker
    now situation is that peoples doesn’t believe that real hackers and carder scammer exists.
    Anyone want to make deal with me any type am available but first
    I‘ll show the proof that am real then make a deal like

    Available Services

    ..Wire Bank Transfer all over the world

    ..Western Union Transfer all over the world

    ..Credit Cards (USA, UK, AUS, CAN, NZ)

    ..School Grade upgrade / remove Records

    ..Spamming Tool

    ..keyloggers / rats

    ..Social Media recovery

    .. Teaching Hacking / spamming / carding (1/2 hours course)

    discount for re-seller

    Contact: 24/7

    fixitrogers@gmail.com

    ReplyDelete

New Blog Location!

I've moved!  I've really enjoyed using Blogger for my blog, but it didn't integrate with my website in the way I wanted.  So I&#...