Testing is the heart of DevOps. In our last blog post, we talked about DevOps – vocabulary, building relationships, and the deployment pipeline. This month we get more into the testing aspect of DevOps.
Test automation is a big part of being able to release consistently and often, but there is so much more to delivering small chunks of value frequently. When a change is committed to your team’s source code repository, think about what steps it goes through to get to production. These steps form your deployment pipeline. Some may not yet be automated, and some may not be appropriate for automation. In the simple diagram below, we show some of the tasks that might require human intervention.
We like to use the term “human-centric” for the testing activities that a person performs. Some of these could potentially be fully automated, such as deploying to a test environment. A team may prefer to leave some decisions such as pushing the deploy button up to a human. Some testing activities, such as exploratory testing, need our human brains, senses and intuition, although we may use tools to assist us such as recorders or data generation scripts. It’s important to visualize the whole deployment pipeline, including the human-centric activities.
The holistic testing loop
Testing in DevOps starts at the very beginning of a new feature – when it’s first identified. You can check out Janet’s blog post on holistic testing to see how she’s adapted the infinite loop for testing specific activities. In this post, we concentrate on the right-hand side of the loop below – the part in yellow.
Test Suite Canvas
When we automate our tests to speed up the deployment pipeline, we should consider the best way to split up the tests, so they run effectively. Ashley Hunsberger has developed a Test Suite Canvas which we have found to be extremely useful when looking at different test suites to know what they are used for, their benefits, who takes responsibility for investigating failures and maintaining tests, and more. Ashley has written about how to use this canvas in this article about feedback in deployment pipelines. One of the most overlooked areas is the data. Do you know how or where you are getting it from, and how is it managed?
Once you know what your automated test suites are, decide the environment where they should run. For example, unit tests should run locally on every developer’s local machine before check-in, and then run automatically on each build after each new commit. They likely won’t run again until the next check-in. A smoke test suite will run on the first deployment to a development environment to make sure that the most important features work correctly, and perhaps someone can log into the system. The smoke test suite might run on any test environment with a new build or run regularly on a staging environment or even on production as a health check if it can run safely. Take care to ensure that each automated test uses reliable test data that isn’t changed by other testing.
Testing quality attributes
At the beginning of the holistic testing loop, the team identifies risks and quality attributes that need to be tested. During development, programmers design the code for ease of testing. For example, they may put in hooks to help test loading time of a page. However, testing quality attributes often cannot be completed until after the code has been written and deployed to a test environment. Accessibility testing, for example, can have some automated components, but ultimately, it means a human being making sure they can access the information.
Your team likely has (or will have, as you proceed along your automation journey) other automated test suites which might include performance, load, and security tests. There also might be human-centric exploratory testing that supplements these automation efforts.
Your team may also have determined that recoverability is important, so you’ll have a playbook to try many different scenarios. Some scenarios may be automated, but many will need human interaction.
There are too many quality attributes to address in a blog post, and each team’s context is different. The important thing is to remember to have to conversation at the beginning to determine your constraints and talk about how to test with each and every story and with every feature.
Testing in production
Testing in production is becoming more common now that it can often be done safely, using techniques such as feature toggles, canary releases, and blue/green deploys. If this isn’t an option (some business domains, for example, do not permit any of these techniques for security reasons), teams can closely monitor and observe production usage and use that information to guide development and testing.
Some companies practice chaos engineering which was pioneered by Netflix. Wikipedia describes it as the discipline of experimenting on a software system in production to build confidence in the system's capability to withstand turbulent and unexpected conditions. This type of testing in production requires a robust infrastructure and fail safes. Chaos engineering can also be conducted on staging environments, providing useful information with lower risk.
We want to do all the testing activities that we can in our test and staging environments, although we can never replicate production usage in those environments. Testing safely in production, and learning from close study of production use, are key for the complicated, multi-service, distributed systems that so many organizations need today. One of our go-to resources for testing in DevOps is Katrina Clokie’s book “Practical Guide to Testing in DevOps”. Lisa’s website has a list of additional books, videos, blog posts and more for testing in DevOps. With skills that enable us to ask good questions, identify and analyze risks, and explore, we testers add so much value on the right side of the DevOps loop. We can help our teams take full advantage of these vital testing activities.