Book Review: Continuous Testing for DevOps Professionals

For this month's book review, I read Continuous Testing for DevOps Professionals: A Practical Guide from Industry Experts, by various authors and edited by Eran Kinsbruner.  The book is divided into four sections: Fundamentals of Continuous Testing, Continuous Testing for Web Apps, Continuous Testing for Mobile Apps, and The Future of Continuous Testing.

The Fundamentals of Continuous Testing section was my favorite, because it focused the most on developing a good Continuous Testing strategy and the elements required.  In Continuous Testing for Web Apps, strategies for testing Responsive Web Applications (RWAs) and Progressive Web Applications (PWAs) were discussed, along with cross-browser testing strategies.  In Continuous Testing for Mobile Apps, chapters included strategies for testing React Native apps and chatbots, as well as tips for using tools like Appium, Espresso, and XCUITest.  Finally, The Future of Continuous Testing took a look at the uses of AI for continuous testing, as well as strategies for testing IoT-enabled devices and Over-the-Top devices. 

Since this book obviously covered a lot of ground, I'll focus on my favorite section, Fundamentals of Continuous Testing.  Contributor Yoram Mizrachi says there are three types of automated testing failures: test code issues; test lab problems, such as an unstable test environment; and execution problems, such as not enough platforms available to run the tests.  There has been much written about solving test code issues, but not enough about solving environment and execution problems, so I was happy to see the suggestions in this book.  To solve environment problems, Brad Johnson suggests using containers such as Docker and Kubernetes to spin up environments for testing.  Because these environments are temporary, they can be completely controlled in terms of data and application state, so there's less chance of test failures due to environment problems.  And Genady Rashkovan offers a solution for execution problems through setting up an automatic detection system for system failures.  After gathering initial data, this detection system can be programmed to predict when failures are about to happen, and execute an automatic reboot or spin up a new VM to mitigate a failure before it happens. 

I also found Tzvika Shahaf's chapter on using smart reporting very insightful.  He notes that test data reporting is often siloed: reports on UI tests use a different format from the reports on API tests, which are in turn different from the reports on performance tests, and so on.  This makes it very difficult for managers to get a sense of the health of the application.  Shahaf recommends creating a unified report for all tests using this process: tag events so they can be easily identified, normalize the test data so it can be used by a single report, correlate events so similar tests are grouped together, and finally display the events with relevant artifacts.  He advises reducing the noise of defects by determining what the most common causes are for test failures and removing the failures that are false negatives.  For example, a test failure that was caused by the test environment going down does not actually indicate that something has gone wrong with the software, so a test report designed to show whether new code is working correctly doesn't need to display those failures.  

I recommend Continuous Testing for DevOps Professionals for anyone who is working on creating a continuous testing system for their application.  There are suggestions for test automation strategies, solving common mobile automation problems, testing connected devices, creating reliable test data, and much more.  My one complaint about the book was that the Kindle version was formatted poorly: the chapter divisions were unclear, there were often footnotes in the middle of the page, and diagrams were broken into pieces over two or more pages.  For that reason, you may want to purchase a paper copy of the book.  But in spite of these problems, I found the book to be very valuable.  

Debugging for Testers

Wikipedia defines debugging as "the process of finding and resolving defects or problems within a computer program that prevent correct operation of computer software or a system".  Often we think of debugging as something that only developers need to do, but this isn't the case.  Here are two reasons why:  first, investigating the cause of a bug when we find it can help the developer fix it faster. Second, since we write automation code ourselves, and since we want to write code that is of high quality just as developers do, we ought to know how to debug our code.  

Let's take a look at three different strategies we can employ when debugging code.

Console output:
Code that is executing in a browser or on a device generally outputs some information to the console.  You can easily see this by opening up Developer Tools in Chrome or the Web Console in Firefox.  When something goes wrong in your application, you can look for error messages in the console.  Helpful error messages like "The file 'address.js' was not found" can tell you exactly what's going wrong.  

Often an error in an application will produce a stack trace.  A stack trace is simply a series of error statements that go in order from the most recent file that was called all the way back to the first file that was called.  Here's a very simple example: let's say that you have a Node application that displays cat photos.  Your main app.js page calls a function called getCats which will load the user's cat photos.  But something goes wrong with getCats, and the application crashes.  Your stack trace might look something like this:

        Error: cannot find photos
        at getCats.js 10:57
        at app.js 15:16
        at internal/main/run_main_module.js:17:47

• The first line of the stack trace is the error- the main cause of what went wrong.  
• The next line shows the last thing that happened before the app crashed: the code was executing in getCats.js, and when it got to line 10, column 57, it couldn't find the photos.  
• The third line shows which file called getCats.js: it was app.js, and it called getCats at line 15, column 16.  
• The final line shows what file was called to run app.js in the first place: an internal Node file that called app.js at line 17, column 47. 

Stack traces are often longer, harder to read, and more complicated than this example, but the more you practice looking at them, the better you will get at finding the most important information.

Logging:
Much of what you see in the console output can be called logging, but there are often specific log entries set up in an application's code that record everything that happens in the application.  I'm fortunate to work with great developers who are adept at creating clear log statements that make it easy to figure out what happened when things go wrong.

Log statements often come with different levels of importance, such as Error, Warning, Info, and Debug.  An application can sometimes be set to only log certain levels of statement.  For example, a Production version of an application might be set to only log Errors and Warnings.  When you're investigating a bug, it may be possible to increase the verbosity of the logs so you can see the Info and Debug statements as well.

You can also make your own log statements, simply by writing code that will output information to the console.  I do this when I'm checking to make sure that my automation code is working like I'm expecting it to.  For example, if I had a do-while statement like this:

do {
     counter++
}
while (counter < 10)

I might add a logging statement that tells me the value of counter as my program progresses:

do {
     console.log ("The value of counter right now is: " + counter)
     counter++
}
while (counter < 10)

The great thing about creating your own log statements is that you can set them up in a way that makes the most sense to you.

Breakpoints:
A breakpoint is a place that you set in the code that will cause the program to pause.  Software often executes very quickly and it can be hard to figure out what's happening as you're flying through the lines of code.  When you set a breakpoint, you can take a look at exactly what all your variable values are at that point in the program.  You can also step through the code slowly to see what happens at each line.

Debuggers are generally available in any language you can write code in.  Here are some examples:

• Python uses the pdb library
• Javascript uses the debugger statement
• Powershell uses breakpoints in the Powershell ISE
• C# has all kinds of debugging tools in Visual Studio
• Java has debugging tools in Eclipse and IntelliJ

I hope this post helps you get started with both debugging your code, and investigating someone else's bugs!

The Joy of JWTs

Have you ever used a JWT before?  If you have tested anything with authentication or authorization, chances are that you have!  The term JWT is pronounced "jot" and it stands for JSON Web Token.  JWTs are created by a company called Auth0, and their purpose is to provide a method for an application to determine whether a user has the credentials necessary to request an asset.  Why are JWTs so great?  Because they allow an application to check for authorization without passing in a username and password or a cookie.  Requests of all kinds can be intercepted, but a JWT contains non-sensitive data and is encrypted, so intercepting it doesn't provide much useful information.  (For more information about the difference between tokens and cookies, see this post.)  Let's learn about how JWTs are made!

A JWT has three parts, which are made up of a series of letters and numbers and are separated by periods.  One of the best ways to learn about JWTs is to practice using the official JWT Debugger, so go to jwt.io and scroll down until you see the Debugger section.

Part One: Header
The header lists the algorithm that is used for encrypting the JWT, and also lists the token type (which is JWT, of course):
{
  "alg": "HS256",
  "typ": "JWT"
}

Part Two: Payload
The payload lists the claims that the user has.  There are three types of claims:
Registered claims: These are standard claims that are predefined by the JWT code, and they include:
     iss (issuer)- who is issuing the claim
     iat (issued at)- what time, in Epoch time, the claim was issued
     exp (expiration time)- what time, in Epoch time, the claim will expire
     aud (audience)- the recipient of the token
     sub (subject)- what kinds of things the recipient can ask for
Public claims: These are other frequently-used claims, and they are added to the JWT registry.  Some examples are name, email, and timezone.
Private claims: These are claims that are defined by the creators of an application, and they are specific to that company.  For example, a company might assign a specific userId to each of their users, and that could be included as a claim.

Here's an example used in the jwt.io Debugger:
{
  "sub": "1234567890",
  "name": "John Doe",
  "iat": 1516239022
}

Here the subject is 1234567890 (which isn't a very descriptive asset), the name of the user who has access to the subject is John Doe, and the token was issued at 1516239022 Epoch time.  Wondering what that time means?  You can use this Epoch time converter to find out!

Part Three: Signature
The signature takes the first two sections and encodes them in Base64.  Then it takes those encoded sections and adds a secret key, which is a long string of letters and numbers.  Finally it encrypts the entire thing with the HMAC SHA256 algorithm.  See my post from last week to understand more about encoding and encryption.

Putting It All Together
The JWT is comprised of the encoded Header, then a period, the encoded Payload, then another period, and finally the encrypted signature.  The JWT Debugger helpfully color-codes these three sections so you can distinguish them.

If you use JWTs regularly in the software you test, try taking one and putting it in the JWT Debugger.  The decoded payload will give you insight into how your application works.

If you don't have a JWT to decode, try making your own!  You can paste values like this into the Payload section of the Debugger and see how the encrypted JWT changes:
{
     "sub": "userData",
     "userName": "kjackvony",
     "iss": 1516239022,
     "exp": 1586606340
}

When you decode a real JWT, the signature doesn't decrypt.  That's because the secret used is a secret!  But because the first and second parts of the JWT are encoded rather than encrypted, they can be decoded.

Using JWTs
How JWTs are used will vary, but a common usage is to pass them with an API request using a Bearer token.  In Postman, that will look something like this:

Testing JWTs
Now that you know all about JWTs, how can you test them?

• Try whatever request you are making without a JWT, to validate that data is not returned.  
• Change or remove one letter in the JWT and make sure that data is not returned when the JWT is used in a request.
• Decode a valid JWT in the Debugger, change it to have different values, and then see if the JWT will work in your request.  
• Use a JWT without a valid signature and make sure that you don't get data in the response.  
• Make note of when the JWT expires, and try a request after it expires to make sure that you don't get data back.  
• Create a JWT that has an issue time of somewhere in the future and make sure that you don't get data back when you use it in your request.
• Decode a JWT and make sure that there is no sensitive information, such as a bank account number, in the Payload.  

Have fun, and happy testing!

New Course! Postman Essential Training

BIG NEWS!  My LinkedIn Learning course on Postman is now live!  This course is an introduction to creating API requests and assertions with Postman.  You'll learn how to create a test collection, run it from the command line, and set it to run as an automated job in Jenkins. 

You can access the course here:  https://www.linkedin.com/learning/postman-essential-training

Encryption and Encoding

We've all encountered mysterious hashed passwords and encrypted texts.  We've heard mysterious terms like "salted" and "SHA256" and wondered what they meant.  This week I decided it was finally time for me to learn about encryption!

The first distinction we need to learn is the difference between encryption and encoding.  Encoding simply means transforming data into a form that's easier to transfer.  URL encoding is a simple type of encoding.  Here's an example: the Coderbyte website has a challenge called "Binary Reversal".  The URL for the page is  https://coderbyte.com/information/Binary%20Reversal; the space between "Binary" and "Reversal" is replaced with "%20".  There are other symbols, such as !, that are replaced in URL encoding as well.  If you'd like to learn more about URL encoding, you can play around with an encoding/decoding tool such as this one.

Another common type of encoding is Base64 encoding.  Base64 encoding is often used to send data; the encoding keeps the bytes from getting corrupted.  This type of encoding is also used in Basic authentication.  You may have seen a username and password encoded in this way when you've logged into a website.  It's important to know that Basic authentication is not secure!  Let's say a malicious actor has intercepted my login with Basic auth, and they've grabbed the authentication string: a2phY2t2b255OnBhc3N3b3JkMTIz.  That looks pretty secure, right?  Nope!  All the hacker needs to do is go to a site like this and decode my username and password.  Try it for yourself!

Now that we know the difference between encoding and encryption, and we know that encoding is not secure, let's learn about encryption.  Encryption transforms data in order to keep it secret.  

A common method of password encryption is hashing, which is a mathematical way of encrypting that is impossible to decrypt.  This seems puzzling- if a string is impossible to decrypt, how will an application ever know that a user's password is correct?  What happens is that the hashed password is saved in the application's authentication database.  When a user logs in, their submitted password is encrypted with the same hashing algorithm that was used to store the password.  If the hashed passwords match, then the password is correct.

What about if two users have the same password?  If a user somehow was able to access the authentication database to view the hashed passwords and they saw that another user had the same hashed password as they did, that user would now know someone else's password.  We solve this problem through salting.  A salt is a short string that is added to the end of a user's password before it is encrypted.  Each password has a different salt added to it, and that salt is saved in the database along with the hashed password.  This way if a hacker gets the list of stored passwords, they won't be able to find any two that are the same.  

A common hashing algorithm is SHA256.  SHA stands for "Secure Hash Algorithm".  The 256 value refers to the number of bits used in the encoding.  

There are other types of encryption that can be decoded.  Two examples are AES encryption and RSA encryption.  AES stands for Advanced Encryption Standard.  This type of encryption is called symmetric key encryption. In symmetric key encryption, the data is encoded with a key, and the receiver of the data needs to have the same key to decrypt the data.  AES encryption is commonly used to transfer data over a VPN.  

RSA stands for Rivest-Shamir-Adleman, who are the three inventors of this encryption method.  RSA uses asymmetric encryption, also called public key encryption, where there is a public key to encode the data and a private key to decode it.  This can work in couple of ways: if the sender of the message knows the receiver's public key, they can encrypt the message and send it; then the receiver decrypts the message with the private key.  Or the sender of the message can sign the message with their private key, and then the receiver of the message can decode it with the sender's public key.  In the second example, the private key is used to show that the message is authentic.  How does the receiver know that the message is authentic if they don't know what the private key is?  They know because if the private key is tampered with, it will be flagged to show that it has been manipulated.  A very common use of RSA encryption is TSL, which is what is used to send data to and from websites.  I wrote about TSL in this post if you'd like to learn more.  

Encryption involves very complicated mathematical algorithms.  Fortunately, we don't have to learn them to understand how encryption works!  In next week's post, I'll talk about how encoding and encryption are used in JWTs.