Yesterday, a LinkedIn connection started a conversation with Janet. He wanted to learn about agile and how to start. It started Janet thinking so we decided to look at the question more deeply since it’s not an easy one to answer, and testing is such a big part of how we think about agile development.
We used the term “agile testing” for many years because it was appropriate at the time. It seemed a natural outgrowth from the “agile” manifesto that came out of the Snowbird gathering in 2001. Agile development focuses on quality, but many agile practices don’t emphasize testing specifically. We believe the agile values, principles and practices apply to testing as well as coding and other development activities.
People frequently ask what “agile testing” is. With a lot of community input, we posted our definition of agile testing few years ago:
- Collaborative testing practices that occur continuously, from inception to delivery and beyond, supporting frequent delivery of value for our customers. Testing activities focus on building quality into the product, using fast feedback loops to validate our understanding. The practices strengthen and support the idea of whole team responsibility for quality.
With the increased visibility of continuous delivery and DevOps culture, more teams, including ones we have worked on and with, have invested in testing activities on the right-hand side of the “DevOps loop”. Dan Ashby captured this so well in his “Continuous Testing in DevOps” post, and we have had interesting discussions with Dan as well as other leading practitioners about the many testing activities throughout the infinite loop of software development.
We came up with the term – “holistic testing”, and Janet wrote a blog post explaining this concept, with a model to illustrate it. Basically, it comes down to this:
- When we test, we need to consider all types of testing, not only the ones we think a tester is responsible for. It includes automation, exploratory testing, or any other type of human-centric testing. It involves the whole team, the product organization, and even the customer. We need to consider testing from a holistic point of view.
Since then, we've published our book, Holistic Testing: Weave Quality Into Your Product, to explain the ideas in detail.
How is holistic testing different from agile testing? There is a huge amount of overlap. Both approaches emphasize fast feedback loops. Both embrace whole-team responsibility for quality and testing.
However, we do see some differences in the holistic approach. Agile testing certainly encompasses the entire software development loop, yet, when we talked about agile testing, we tended to talk more about the left-hand side of the loop. We put a lot of emphasis on building shared understanding of features, testing feature ideas, guiding development with technology- and business-facing tests. We also emphasized continuous improvement, using practices such as retrospectives to identify obstacles and design experiments to move forward.
Holistic testing balances testing activities on both sides of the DevOps loop and shows how testing needs to be a team activity. Many organizations still leave the right-hand side of the loop up to operations specialists and site reliability engineers. This is because many testers still lack knowledge of how code is instrumented to store structured data in logs, or they may not know how to use any of the monitoring and observability tools available today. Anyone on the team can take advantage of analytics tools that show what our production users are doing. We need testing on both sides of the loop to continuously deliver value for our customers at a sustainable pace.
The Agile Manifesto is more than 25 years old, and the term “agile” has gained a lot of unfair baggage over the years. Some organizations implement a development framework and say they are agile, even though they still deliver poor quality software infrequently. Labels are hard. “Holistic” is descriptive, it is more precise. It reflects the whole-team approach. It encompasses the whole development cycle. Words are important, and we believe “holistic testing” is a better way to convey this approach to building quality in.