Moving up the list in our series of blog posts about key success factors for agile testing, we get to automation as # 3. We include automation in testing as one of our success factors because we believe you cannot deliver changes frequently, at a sustainable pace, without it. Your regression test suite (when automated) is a change detector. If a test fails, it signifies something is wrong – it could be the test, it could be the code has changed and you forgot to change the test. Or perhaps another part of the system was affected, and you didn’t realize it was even impacted by the change. Whatever the reason, team members need to analyze the failure and determine the cause.
Automation has many flavors. As more teams work towards continuous delivery, it has become even more important.
Expanding your strategy
Most of you will have seen some variation of the test automation pyramid, originally created by Mike Cohn in the early 2000s. This model still works well as a thinking tool – especially if you are new to automation. We use it as a starting point to talk with the team about at what level should we automate specific tests. The conversation is the important part – the automation becomes a secondary but necessary task.
A conversation might sound like this:
Scrum master: Let’s talk about what testing we might need for this feature, and what kind of automation we need to plan for.
Tester: Well, if we look at the different scenarios from the flow diagram we drew, we will definitely need some full work-flow tests. Let’s draw the pyramid and see what else we need.
Programmer: We can test x, y and z at the unit test level, and all the variations.
Tester: What about the error conditions? Where would they fall? I do need to see they are on the screen but don’t want to test every variation.
Programmer: We can help there by …….
The variation of the pyramid shown here relates only to functional tests. In both More Agile Testing, and Agile Testing Condensed, we included other variations. Lisa covered some more in a blog post series on modeling your test automation strategy. We think it is important for teams to experiment and think about the visual model that helps them design an automation strategy that fits their context.
There are many other kinds of automation that needs to be considered as well. To succeed with continuous delivery, teams need to automate as many stages as possible in the path their code takes to get to production. This includes continuous integration (CI), creating test environments in the cloud, and deploying artifacts to various environments, In addition, automation is needed to develop useful information for monitoring dashboards and alerts.
Our automation toolbox is ever-expanding. One team Lisa worked on, discussed the possibility of using Slack APIs to trigger creation of complex test environments from Slack, as other teams have done. Modern test automation frameworks and libraries have features like “auto-healing” UI-level tests so they adapt to minor UI changes. Many tools use machine learning to detect visual and performance anomalies. New technology helps us keep pace with the increased complexity of today’s systems.
Enabling the whole team to mitigate risks
Perhaps most importantly, automating everything that can be automated frees our time to focus on human-centric testing activities, where we need our brains, problem-solving skills and creativity. We can take time to ’think outside the box’ and discover the ‘unknown unknowns’ we can’t identify as we plan and build new features.
Several years of State of DevOps Survey results, reported in the book Accelerate by Dr. Nicole Forsgren, Jez Humble and Gene Kim, back up what we have observed for a couple of decades now. Having fast, reliable automated tests correlates with high team performance. The automated tests are fast and reliable when the developers own them, run them locally, investigate the failures in the CI test suites. The book makes the important point – this does not mean we don’t need testers! In these high-performing teams, testers and developers work together to automate the tests, and the testers lead the human-centric testing activities like exploratory testing. We were happy when we learned that science backs up what we have known – the whole team approach to testing and quality really does work.