Email Subscription Form

Saturday, October 12, 2019

Why You Should Be Testing in Production

This is a true story; I'm keeping the details vague to protect those involved.  Once there was a software team that was implementing new functionality.  They tested the new functionality in their QA environment, and it worked just fine.  So they scheduled a deployment: first to the Staging environment, then to Production.  They didn't have any automated tests for the new feature, because it was tricky to automate.  And they didn't bother to do any manual tests in Staging or Production, reasoning that if it worked in the QA environment, it must work everywhere.

You can probably guess what happened next- they started getting calls from customers that the new feature didn't work.  They investigated and found that this was true.  Then they tried out the feature in the Staging environment and found that it didn't work there either.  As it turned out, the team had used hard-coded configuration strings that were only valid in the QA environment.  If they had simply done ONE test in the Staging or Production environment, they would have noticed that something was wrong.  Instead, it was left to the customers to notice the problem.

There are two main reasons why things that work in a QA environment don't work in a Production environment:

1) Configuration problems- This is what happened with the team described above.  Software is complicated, and there are often multiple servers and databases that need to talk to each other in order for the software to work properly.  Keeping software secure means that each part of the application needs to be protected by passwords or other configuration strings.  If any one of those strings is incorrect, the software won't work completely.

2) Deployment problems- In this age of microservices, deploying software usually means deploying several different APIs.  In a large organization, there may be different teams responsible for different APIs.  For example, when a new feature in API A needs the new code in API B to work properly, API B will need to be deployed first.  It's possible that Team B will forget to deploy API B or not even realize that it needs to be deployed.  In cases like this, Team A might assume that API B had been deployed, and they will go ahead and deploy API A.  Without testing, Team A will have no way of knowing that the new feature isn't working.

By running tests in every environment, you can quickly discover if you have configuration or deployment problems.  It's often not necessary to go through extensive testing of a new feature in Production if you've already tested it in QA, but it is vital that you do at least SOME testing to verify that it's working!  We never want to have our customers find problems before we do.


  1. It's worth clarifying:
    - testing in production is in addition to testing that you would do?
    - testing in production is not a substitute for testing pre-release?

    Also worth thinking about architectural changes that aren't simple. In that case you can't just roll back once the feature is released.

    1. Hi Nilan- Yes! You should always do extensive testing in Pre-Prod environments. Testing in Prod is NOT a substitute for testing in QA. And whenever you go to Prod, you should have a rollback or mitigation strategy in place if something goes wrong. Thanks for these excellent points!

  2. This comment has been removed by the author.

  3. Hi Kristin, Thanks for the blog posts.
    I have question on this, so we tester's test and validate the release in test environment, but we are not involved after it has been released to production. So who will test the

    1. Hi Bharath- yes, you should be testing in pre-production and production! If this is not currently allowed at your company, share this article with your manager and see if you can get at least some access to every environment.

  4. This comment has been removed by the author.


Book Review: Perfect Software and Other Illusions About Testing

"Perfect Software and Other Illusions About Testing", by Gerald Weinberg, is the best book on testing I have ever read.  It is a m...