We used the term “agile testing” for many years because it was appropriate at the time. It seemed a natural outgrowth from the “agile” manifesto that came out of the Snowbird gathering in 2001. Agile development focuses on quality, but many agile practices don’t emphasize testing specifically. We believe the agile values, principles and practices apply to testing as well as coding and other development activities.

People frequently ask what “agile testing” is. With a lot of community input, we posted our definition of agile testing few years ago:

• Collaborative testing practices that occur continuously, from inception to delivery and beyond, supporting frequent delivery of value for our customers. Testing activities focus on building quality into the product, using fast feedback loops to validate our understanding. The practices strengthen and support the idea of whole team responsibility for quality.

With the increased visibility of continuous delivery and DevOps culture, more teams, including ones we have worked on and with, have invested in testing activities on the right-hand side of the “DevOps loop”. Dan Ashby captured this so well in his “Continuous Testing in DevOps” post, and we have had interesting discussions with Dan as well as other leading practitioners about the many testing activities throughout the infinite loop of software development.

We came up with the term – “holistic testing”, and Janet wrote a blog post explaining this concept, with a model to illustrate it. Basically, it comes down to this:

• When we test, we need to consider all types of testing, not only the ones we think a tester is responsible for. It includes automation, exploratory testing, or any other type of human-centric testing. It involves the whole team, the product organization, and even the customer. We need to consider testing from a holistic point of view.

Since then, we've published our book, Holistic Testing: Weave Quality Into Your Product, to explain the ideas in detail.

How is holistic testing different from agile testing? There is a huge amount of overlap. Both approaches emphasize fast feedback loops. Both embrace whole-team responsibility for quality and testing.

However, we do see some differences in the holistic approach. Agile testing certainly encompasses the entire software development loop, yet, when we talked about agile testing, we tended to talk more about the left-hand side of the loop. We put a lot of emphasis on building shared understanding of features, testing feature ideas, guiding development with technology- and business-facing tests. We also emphasized continuous improvement, using practices such as retrospectives to identify obstacles and design experiments to move forward.

Holistic testing balances testing activities on both sides of the DevOps loop and shows how testing needs to be a team activity. Many organizations still leave the right-hand side of the loop up to operations specialists and site reliability engineers. This is because many testers still lack knowledge of how code is instrumented to store structured data in logs, or they may not know how to use any of the monitoring and observability tools available today. Anyone on the team can take advantage of analytics tools that show what our production users are doing. We need testing on both sides of the loop to continuously deliver value for our customers at a sustainable pace.

The Agile Manifesto is more than 25 years old, and the term “agile” has gained a lot of unfair baggage over the years. Some organizations implement a development framework and say they are agile, even though they still deliver poor quality software infrequently. Labels are hard. “Holistic” is descriptive, it is more precise. It reflects the whole-team approach. It encompasses the whole development cycle. Words are important, and we believe “holistic testing” is a better way to convey this approach to building quality in.

User Acceptance Testing (UAT) is important in large customized applications, as well as internal applications. It’s performed by all affected business departments to verify usability of the system and to confirm existing and new (emphasis on new) business functionality of the system. Your customers are the ones who must live with the application, so they need to make sure it works on their system and with their data.

Teams that still perform UAT, likely do not have continuous delivery in place. However, that is not a rule and some clients still want some form of final say before they accept a new release.

In one team that Lisa worked in, prior to a deploy to production, they first installed the release candidate in the pre-production environment. Lisa met with the client’s test manager and explained all the testing already completed – automated tests at the unit and UI level, acceptance testing, and exploratory testing. This was a totally new idea to them, and they were thrilled that they could cut way down on the UAT. Since several different teams also delivered changes to the system, any issue found was reported during a conference call and the appropriate team took charge of it. They had working agreements with the customer about what would constitute a critical show-stopper bug that they had to fix and what could be turned into a user story and put off to the next release.

Janet often tells a story of a team that had three-month release cycles (quarterly). The day she joined, they had just put the release into the customer’s test environment for UAT. She was told it would take six weeks. What?  How could that be? It’s half-way through the next release cycle. Crazy. Slowly she started introducing the customer representative to features earlier and earlier, and on the second release cycle, when the team delivered the candidate for UAT, the customer quoted one day to test in her environment. Mission accomplished.

Fitting UAT into the Holistic Testing Model

The key is to think about how to get testing moved as early in the cycle as possible. Consider what are the constraints, and how can you remove or mitigate them. Have conversations with your customers in advance to see how much UAT will be necessary and how any bugs found will be handled.

We have not specified UAT as an example of a test in our Holistic Testing Model. Many teams no longer have the idea of a separate UAT if they are collaborating very closely with their customers. Or, they have a less formal UAT if they are practicing continuous delivery and using a release strategy such as feature flags to hide new changes until they have been tested by their customers.

If we added UAT as an example in the Holistic Testing Model, it would fit into the Deploy stage, but that doesn’t mean you have to wait until right before a production release to have customers test the system.

Embracing today’s leading practices for deploying, monitoring, and observing the system in production can mitigate risk by allowing instant response to production issues. Teams have many options. Collaborate with your customers to see what fits best for your situation.

Exploring a feature together in an ensemble format is a great way to spot those hard-to-find issues and learn what might be missing. In an ensemble, people rotate through the roles of navigator and driver, with the navigator telling the driver what to type, and everyone in the ensemble is free to contribute their ideas. A short time box is set, and roles rotate at the end of each one.

The first time you try something new like ensemble testing (or even pairing), it will feel awkward and contrived. The key to getting better at it is deliberate practice. Start small, perhaps with only three people, and rotate, each getting familiar with the roles. Practice what it feels like. The more you practice, the more comfortable you’ll be. At that point you know the benefits and can share what you’ve learned and can expand it to include others.

In various organizations where she worked, Lisa and her fellow testers/quality engineers hosted a weekly ensemble testing session, open to anyone who wanted to join. Sometimes these were focused 30-minute sessions, sometimes an hour or 90 minutes for deeper exploration. Programmers on all the feature teams were invited to book a session to have the group explore their latest new features.

One way to organize these sessions is to have at least one programmer from the feature team join in to answer questions and benefit from the quick feedback. The programmer and the quality engineer who is facilitating the session, collaborate to prepare a test charter in advance. This charter guides the ensemble on what to test and provides links to the build that is to be tested.

During the session, each driver shares their screen so the others can follow along. These sessions have proven so useful that additional sessions were scheduled and some engineering teams organized their own ensemble testing sessions.

Another ensemble testing technique Lisa’s teams have used is a similar format, aimed at acceptance testing stories. As soon as a team had several stories “finished”, they scheduled a 30-minute ensemble testing session that included the product owner, the designer, a tester, the programmers who worked on the stories, and someone from customer support. Rather than switch roles on a time schedule, they switched for each story. Any questions, whether related to design, functionality, or impact on customers, someone in the session was able to answer. 

Ensemble testing brings many advantages. People with different specialized skills working together in this type of group format can overcome unconscious biases and blind spots and achieve more in a short amount of time than if they worked individually. The obvious benefits encourage everyone to be more willing to collaborate real-time in pairs or groups. So, if you are feeling timid to bring this idea up with your team, practice deliberately to understand how, and then you will be more confident to bring this powerful technique to your team.

Our work together is a case in point. After we joined our first Extreme Programming teams, we shared ideas and practices from each of our teams and evolved new ideas together. We also shared these with other practitioners, who added their own insights.  

For example, back in 2003 Janet had a conversation with Brian Marick and Bret Pettichord at the end of a long day of a conference. Brian was explaining this idea he had for a taxonomy of testing types for agile development. After many questions and much conversation, this taxonomy grew into Brian’s agile testing matrix.

We both used the matrix with our own teams and evolved our own version which we used (with Brian’s permission) in our first book Agile Testing: A Practical Guide for Testers and Agile Teams. Not only have we continued to evolve our quadrants model, but other people have created their own adaptions (you can see some of them in Chapter 8 of More Agile Testing, which you can download from our website.)

More recently, Dan Ashby created a new model he’d evolved for “left” and “right” testing cycles. He said it was inspired by a shift left – shift right testing diagram Janet created. Janet’s model (below) was inspired by Dan’s original continuous testing blog post and model.

The resulting Twitter conversation spawned a meeting between Janet, Dan and Rob Meaney to clarify and challenge these ideas to create better ones. The outcomes of that discussion has been shared in Janet’s blog posts, and went on to become the Holistic Testing model that we talk about in our downloadable mini book Holistic Testing: Weaving quality into your product. No doubt it will trigger more ideas by other practitioners.

Anne-Marie Charrett has built upon it with her product excellence model. This represents a holistic view of quality across discovery, delivery and support. It focuses more on product quality, rather than  process quality.

These are examples showing the power of sharing ideas with others. When a person has the courage to put their thoughts out in public, inviting conversations, one idea can become something bigger.

Even small conversations can have a huge impact in a team or even on an organization. It starts with one person being brave and sharing their idea – making it visible and inviting others in. Make it grow!

Note: If you want to know more about continuous testing models, you can check out Lisa’s blog post.  https://lisacrispin.com/2020/11/01/shifting-left-right-in-our-continuous-world/)

This is the second in our series introducing some of the highly experienced practitioners who teach our Holistic Testing courses. Some of you are already familiar with Gáspár Nagy. Janet and Lisa had the good fortune to meet Gáspár at testing conferences many years ago. He’s been in the software development business for more than 20 years as an architect and agile developer coach. He’s an author, test tool developer, and a leader in the open-source community. We’re honored to have him on board as an ATF training provider and instructor since 2018.

Q: Gáspár, please tell us your "origin story", how did you get into the software profession?

Like many things, it started with a coincidence. I had to do home schooling when I was 14 because of  surgery. Without even knowing me, my teacher of informatics thought that it must be very boring at home and lent me one of the school computers. This was a big thing in Hungary in 1992. There were no games on the computer (and no internet at that time), so I started coding and have never stopped since then. Sometimes even a small help can cause fundamental changes and I'm very grateful to my teacher for what she did. Now it's my turn and my responsibility to give it forward.

Q: You're a leader for behavior-driven development (BDD), including being the lead developer for SpecFlow, the SpecSync synchronization tool, and writing the BDD book series with Seb Rose. When did you first try BDD? What made you want to try it? Did you have good results right away?

In 2009 we had projects with massive integration-level test automation. We were stunned by the real actual costs of creating and maintaining those tests, so we were seeking ways to increase the value provided by the tests and reducing the maintenance costs. (Sounds familiar?) Somehow our idea was that a better separating of the intent of the tests or the actions performed by the tests from the technical solution would be beneficial. We had tried different approaches already when we found Cucumber that seemed to do exactly what we wanted. The concept of BDD worked for us right from the beginning, or at least we always saw the light at the end of the tunnel, but the new concept introduced or brought other challenges. In the end it took us some time until we could say that we had a working model.

Q: What motivated you to help teams learn about BDD, SpecFlow, test-driven development (TDD), and Extreme Programming (XP) by providing your own range of training courses?

I like to do training. The feeling when you find the right example that illustrates a concept and can explain it in a way that the other person understands it, compares to nothing else. It complements the engineering work I would otherwise do. Through the courses I can connect with many people, learning about their perspectives and challenges. It is also useful to validate my own thoughts. It is very exhaustive, but it is worth.

Q: What made you decide to add our holistic testing courses to your training offerings?

My journey to agile software development started with project management (Scrum), agile engineering techniques and agile requirement management. But I felt that something was missing. When I read the Agile Testing book, I realized that this is the missing link. So basically, your book filled up a white spot on my map. For me, the holistic testing courses are a good way to pass on this "yay, I've found it" feeling. And I feel honored to work with you on this.

Q: We’re so glad you found our book and our courses valuable! You helped us refine and improve the new course, "Holistic Testing for Continuous Delivery". Who do you think can benefit most from it? 

Continuous Delivery (CD) is a good starting point. The goals of CD are easy to understand and the CD pipeline is a concrete asset that makes the concept visible for everyone. But to be able to build up a working CD process, you really need everyone to collaborate and at the end you will learn all important concepts of the thing that we call holistic testing. So, I think this is beneficial for everyone who are on this journey, regardless of whether you will ever configure a pipeline yourself. Continuous Delivery (just like holistic testing) is not only for the big online companies. The concept works for projects in all sizes and nature.

Q: When you build code yourself, such as custom extensions for clients using SpecFlow, do you use a holistic approach? Do you learn things from your own development work that you share in the classes you teach?

I think the short answer would be - yes. But this is a journey and a continuous responsibility. Like many others, I tend to over trust myself and sometimes I feel that "I'm good enough already, I might not need this or that." You can call this laziness or even a risk-based approach, but at the end it is always the same: It works indeed… mostly. However, there may be a circumstance and then a problem is caused that could have been avoided by doing the things "right". "Mostly" is not a good quality measure. But I learn from these incidents and they make the best stories that I can share with my course attendees.

Q: Is there anything else you'd like to share about yourself and your interests?

I got into testing from software development, so I am generally interested in how devs and testers can better collaborate with each other. This is not easy though. Just putting devs and testers into the same team is not enough. We need to keep seeking opportunities and concepts that enable collaboration, so we can really see quality in a holistic way.

Q: Please share any upcoming events and classes you have planned. Also please share links to your website, any writing or videos you have available so people can learn more about your work.

I have some upcoming courses in the first quarter of 2024. In the "BDD Vitals" course you gain essential knowledge and skills for writing better BDD scenarios, which enable you to become a strong member of a BDD team (https://www.specsolutions.eu/events/courses/202403bddvitalsonline/). If you have interest in using SpecFlow and test automation as well, I strongly recommend my "BDD with SpecFlow" course (https://www.specsolutions.eu/events/courses/202403specflowonline/). In both courses the attendees can rely on a lot of exercises and discussions.

And last but not least, in April I am awaiting attendees for the “Holistic Testing For Continuous Delivery for a quality DevOps culture” course, where you can learn ways to apply the infinite loop Holistic Testing model to testing activities that help your team succeed with continuous delivery (https://www.specsolutions.eu/events/courses/202404holistictesting-cd/).

All these courses are offered in remote form. Further information is available at our company website (www.specsolutions.eu).

In software organizations that use a phased-and-gated and siloed process, testers tend to focus on finding bugs after the code has been written. In agile development, our goal is to help deliver valuable software frequently, solving our customers’ problems, by doing whatever we can. Our deep skills may be in testing, and as a tester in an agile team, we’re willing to take on any task to help the team, often collaborating with others. We’re not afraid to join in activities on both sides of the infinite loop of software development.  Our Holistic Testing model visualizes this idea. To us, we think of shift left and right as part of the infinite loop rather than a linear timeline. There are so many places where we need to think about testing.

Having an agile testing mindset means getting out of your comfort zone often. Several years ago, Lisa became interested in observability, the ability to ask questions and learn about your system in production without having to ship new code. Learning the basic concepts, but without yet having in-depth expertise and experience, she saw the importance for testers to get involved. She took a job helping to build an observability practice – without already having the skills. Lisa was way outside her comfort zone in this new job. She built relationships with developers and site reliability engineers who were also keen to adopt it. They told her that she added value by asking questions and bringing in new ideas. They appreciated her perspective as a testing and quality specialist. That’s what a growth mindset is all about – being willing to fail, but willing to learn something new, and help the team in the process.

Janet recalls a time when one of her grandchildren, Jo, was learning to surf behind a boat. She was able to stand up on the board, but every time she looked forward, all she saw was the waves coming towards her and she would fall. Finally, Janet and her sister told her, “Look at us instead.” The next time Jo stood up, her aunt said… “Look in my eyes.  Just keep looking at me.” Because she wasn’t concentrating on all the problems, she surfed for four minutes.

The next time you don’t know how to do something, we encourage you to think before you say ‘no’.. Is there an opportunity for you to stretch yourself and learn something new that will help build your own skills, and also contribute to your team’s success?

Janet Gregory, Lisa Crispin and José Diaz co-founded the Agile Testing Fellowship to help teams and practitioners around the world learn to succeed with testing in agile environments. We agreed from the start to be highly selective about who we would choose as instructors for our courses. All of our trainers are experienced testing practitioners, who are also excellent teachers and facilitators. 

We'd love for more people to know about our awesome instructors and training providers. We'll interview one of them for each post in this new series.

Our first featured instructor is Prathan Dansakulcharoenkit. He started teaching our course, "Holistic Testing: Strategies for Agile Teams", in 2018. He's helped more than 160 students along their agile testing learning journey. We hope you enjoy learning more about Prathan. Many thanks to Prathan for sharing his journey with us!

Q. Hi Prathan, please tell us a little about yourself.

A. My name is Prathan Dansakulcharoenkit. I live in Bangkok, Thailand. I’m running 3 companies in Bangkok and they provide the services of Agile Software Development. In software testing, I have been working for 20 years since 2003.

Q. How did you get started in the software industry? What attracted you to the testing and quality side of things?

A. I started to work in software development in 2003 as a System Administrator and helped the development team to test the software before delivering it to the customer, too. As a full-time software tester, I worked in the number one portal website company in Thailand in 2005 - 2010 and set up the software testing process, tools and acquired the team members. I moved the internal team knowledge base portal website to the public website in the name, WeLoveBug.com in 2008. In 2009, I was promoted to be the service and operation manager and still manage the software testing process and team of the portal website, too.

In 2010, I changed the job to be the software development manager in the Thailand ecommerce website where they joined with the number one ecommerce website from Japan, Rakuten. I set up the software testing process, tools and team. In that time was the first step to adapt the test-first development practice with the Scrum framework and some practices from Extreme Programming.

In 2012, my colleague and I set up the company, Siam Chamnankit Co., Ltd., where we provide the services to change the software development process and practices from the sequence phases e.g. waterfall model to the Agile Software Development with the Scrum Framework, Extreme Programming and relevant practices. The other service is the workshops, software testing is the main workshop we provide for those who work in the software development both the functional tests and non-functional tests.

In 2019, I split the software testing services from Siam Chamnankit Co., Ltd. to the new company name, We Love Bug Co., Ltd. We Love Bug Co., Ltd. provides the all in one software testing services for any software development process.

Q: What’s the biggest benefit you see to a holistic and agile approach to software development and testing? 

A: Preventing the defects before starting development and the whole team approach are the two biggest benefits that I got from my personal experience and delivered the experiences to my colleague, customers, universities, software testing community and software development community in Thailand.

Q: When did you become interested in helping others learn ways to succeed with testing and build quality into software products?

In 2008 was the first move to helping the others by setting up the website to share the knowledge and experiences through the blog. The training and workshops are the second step both inside the company and some public for the external who are interested. The third step is to provide the services to lead change of the test-last development to the test-first development approach for any software development process.

Q: What do you enjoy most about facilitating training courses?

As an experienced giver, the enjoyable moment is when the participants get the AHA moment, nod their head and reflect on what they steal from me every hour.

Q: How did you decide to become a training provider and instructor for Agile Testing Fellowship’s Holistic Testing course?

A: I've been providing the software testing workshops since 2012 and the main source of knowledge and experience are from the Agile Testing: A Practical Guide for Testers and Agile Teams book, and slides by Janet and Lisa that are shared on the internet. My colleague and I practice the knowledge and experience from the book and slides in the projects again and again before bringing them into the workshops to prove for the participants that not just the words in the book and slides but they are real world experience and can adapt in Thailand, too.

In February 2016, My colleague and I attended the Agile Testing workshop in Singapore and it was the first time we met Janet in real life. On the last day of the workshop, I asked Janet about the loyalty program because I need to bring the workshop to Thailand. That was the beginning of becoming a training provider and instructor.

Q: Please share any upcoming events you’re participating in. Also please share links to your website, any writing or videos you have available so people can learn more about your work.

A: For 2024, I’m going to provide the Holistic Testing both Strategies for Agile Teams and Continuous Delivery every quarter in Thailand. The other sources of knowledge sharing are

• Software Testing blog (Thai language): www.welovebug.com
• Scrum in real life blog (Thai language): www.scrum123.com
• YouTube Channel (Thai language): https://www.youtube.com/@siamchamnankit5205

A good next step is to do a quality practices assessment using the QPAM model. Check out the latest episode of our Donkeys & Dragons video chat to find out more. In this episode, Janet switches from co-host to guest! She and Selena Delesie answer Lisa’s questions about their Quality Practices Assessment Model (QPAM).

When you watch this 18 minute episode, you’ll learn how they came up with this new model. They explain how teams can use QPAM to understand where they are in their quality journey. The conversation includes some of the many benefits of QPAM, as well as an overview of their books about it.

The “Assessing Agile Practices with QPAM: Enabling Teams to Improve” ebook is available now on LeanPub.

As the holistic testing infinite loop cycles around the right side and back towards the left, we use the information about production usage to drive changes that will solve customer problems. A significant production outage might be followed up with a retrospective (some people call these postmortems, but we hope nobody died). We recommend that teams use visual collaboration tools as they explore issues like this. Root cause analysis tools such as fishbone or Ishikawa diagrams (also called fishbone diagrams) may be helpful. Learning outcomes can be used in the discovery stage to come up with ideas that address the root problems.

We can learn much more from production usage observations besides outages and system unavailability. For example, analytics can tell us how many customers tried out a new feature. Today’s tools can even show us where customers struggled with a user interface. We can analyze data for our service level indicators to see if objectives and service level agreements were met. Use retrospectives and brainstorming meetings to understand these better and prioritize the most important challenges to address.

A technique that has worked well for Lisa’s team is using “small, frugal experiments”, something Lisa learned from Linda Rising’s  talk at Agile 2015 (you can see the slides, if you are an Agile Alliance member). Once your team has identified the challenge you want to address and dug into the associated issues to understand the problem better, design one or more small experiments to make progress towards that goal or make that problem smaller. These experiments should only last up to a few weeks, so that if they don’t work, you haven’t invested much time and you’ve learned something from them I any case.

We like to use this format for a hypothesis:

      We believe that <an action that we are going to take>

      Will result in <some aspect of progress towards addressing a challenge or achieving a goal>

      We’ll know we have succeeded when <a concrete measurement that will show progress towards the goal>

For example, let’s say our team has seen a high number of 500 errors in production on our web-based application. A 500-error means reduced availability, so our system availability is 99.7% instead of our objective of 99.9%. We investigate further to be sure that the 500 errors were a significant cause of some downtime. We might have a hypothesis such as:

           We believe that instrumenting our code to capture all events leading up to 500 errors

            Will lead to faster diagnosing and fixing 500 errors in production

            We’ll know we have succeeded when our availability metric is up to 99.8% within three weeks

If our availability metric wasn’t increasing within three weeks, then we’d know there might be another cause to our downtime, or we need to focus on preventing the 500 errors instead. We can try more small experiments.

Another example: Our analytics show that customers are slow to try out new features that we release or do not use them at all. If we couldn’t get access to our customers to ask why they weren’t using the new features, as a team, including designers and product owner, we decide to incorporate usability testing before releasing new features. Our hypothesis:

          We believe that collaborating with designers to do usability testing of each new feature before release

           Will lead to more customers trying out new features right away

           We’ll know we have succeeded when 20% of customers try our next new feature within 48 hours of release

There are many ways to learn from production usage and guide future product changes. Getting the whole team, including testers, programmers, designers, product owners and operations specialists, collaborating to learn is key. Whatever method you choose to continually improve, it is important that you have some way to measure progress and know when you have succeeded.

Both monitoring and observability use telemetry (measurements for data collection) from an application’s code to understand production system behavior. However, there are important distinctions between the two.

Monitoring is about looking for behavior that we expect. We collect log data, and then use monitoring tools to aggregate it, analyze it, and produce dashboards and alerts. We compare the actual data with what we expected. Monitoring is about predictable failures; it lets us see deviation from expected behavior.

Observability is about the behavior that we didn’t expect or couldn’t anticipate. Today’s complex systems fail in complex ways. Stuff happens – and we need to ask questions that we didn’t expect to ask. One of the best ways to know if you have observability is to answer this question: “Do you have to add new instrumentation to the code and redeploy it to diagnose a problem?” Practicing observability means being able to diagnose a problem without that step.

We’re borrowing a visual from James Lyndsay with a slight nuance switch to show the difference.

Note: The original Venn diagram is available as a download from https://www.workroom-productions.com/why-exploration-has-a-place-in-any-strategy/. We encourage you to read it.

Many teams have used monitoring tools for many years but find that their customers may still feel pain even as their monitoring dashboards show a healthy picture. Hopefully, we did all the important testing before releasing a change anticipating what might happen in production, and we might even have done chaos engineering to explore different types of failures in a controlled way.

From experience, we know we cannot think of everything, but we know that we have to be ready for anything. That is we capture all the information we possibly can and use specially designed tools to analyze it quickly – to diagnose problems quickly. We like to think of it as an early warning system. That is what we call observability.

In her new Test Automation University course, Introduction to Observability for Test Automation, Abby Bangser includes a terrific summary of the monitoring vs. observability. Some important highlights are:

• With monitoring, we have a way to track, identify and lock in behavior across a wide diversity of systems and we can do this in a fairly standard way.
• Monitoring can consistently alert on changes to the baseline.
• Observability aims at supporting the unknown.
• An observable system is one that can answer new, unique and complex questions without delay.
• Observability provides specifics to triage and creatively explore even highly specific impact bugs.

You can get in-depth information about observability in Abby’s course, including the origin of the term. Observability is one important tool among many to learn how our customers use our product, what pain points they have, what features they might still need. When we build our software features, we need to build in the telemetry for monitoring, observability, and analytics. All of this enables us to respond quickly to production issues, no matter how many or how few users feel the impact.

In this blog post, we look at quality within the ‘understanding’ stage of our holistic testing loop.

Lisa has often had this experience: The development team discusses a story during the iteration planning meeting, gives it an estimate, and starts working on it. They do a great job of coding and testing the story using good practices and delivers it to the product owner for acceptance. The product owner rejects it with a puzzled comment: “This isn’t what I asked for.” Grrr! So frustrating! The development team understood the story one way – the product owner understood it a totally different way. They failed to share the same view of the outcomes.

When teams work together with product folks and others in the discovery and planning activities, they start to build shared understanding. Once the high-level picture is established, it’s time to encapsulate that shared understanding in artifacts that will helps build the right thing. In our experience, there are a few factors to succeed with this effort.

Conversations among people with diverse perspective

In a recent episode of the “Making Tech Better” podcast, Lou Downe referred to Melvin Conway’s work, including this striking thought: “The quality of software is directly related to the quality of conversations we have”. Lou said collaboration is a privilege. Teams need to find ways to have conversations that often circumvent their organizational structures.

For example, the delivery team may use different terminology than people in other teams such as marketing, data and design teams, so the delivery team may need to collaborate with them, making the team knows how a particular feature is valuable to the customer, what it should look like, and how it should behave. There needs to be a safe space to talk with people in other parts of the organization.

Structuring the conversations

Once the team has committed to enabling cross-team collaboration, they can take advantage of the many frameworks and tools that enable them to get the most value from these conversations. The business rules and concrete examples that define how a capability should work can be turned into business-facing tests that guide development.

One practice that forms a great foundation for shared understanding is what we call Power of Three, or as George Dinwiddie dubbed it, Three Amigos. Before each team story planning workshop, a product person, a programmer, and a tester may meet to go over the proposed stories. These days, with our complex systems, we often need more folks, maybe up to four to six – a designer, a data expert, a marketing specialist, an operations specialist – whomever has valuable insight to the stories being discussed.

Keep these discussions quick and focused with a framework such as example mapping by Matt Wynne. Other visual collaboration tools such as mind mapping or simply using virtual sticky notes also help.

It’s important to ask questions. We like to use the “7 Product Dimensions” from Ellen Gottesdiener and Mary Gorman to think of good questions related to different quality attributes – along the lines of, “Where will customers be when they use the product? What interface or device will they use? Do we have test data? How many concurrent users do we need to support?”  Asking these types of questions help to minimize the “unknown unknowns”.
         

Teams should also discuss risks that they can’t fully mitigate through testing. What events should be logged in the new workflow, to allow identifying and responding to issues once the feature is in production? What alerts and dashboards would be helpful? The answers add to shared understanding of the technical implementation.

The key is agreeing on the purpose of each story, its main value to customers, and specifying the business rules, with each rule illustrated with concrete examples. As shared understanding grows, the conversation will also uncover missing stories or ones that are too large.

These conversations may also produce low-fidelity prototypes, flow diagrams and other visuals that can be used to explain the desired outcomes to other team members and stakeholders outside the team. Last week, during a course that Janet facilitated, some of the participants were amazed at how much information they were each missing when they such a visual exercise. Share the outcomes with the rest of the team so everyone starts from the same level of knowledge when you have your team-wide planning discussion.

Guiding development together

Working together, programmers, testers and product owners can turn the business rules and examples from workshops into executable tests, using a domain-specific language that people outside the technology team can also understand and review. These tests form the basis for acceptance-test or behavior-driven development, where the business-facing tests guide development. They help to build the right thing the first time we try.

Using this holistic process of getting a cross-section of perspectives together to build shared understanding and produce artifacts that let us know what to code is a proven way to avoid re-work and waste due to stories getting rejected by the product owner or customer. Teams with strong shared understanding of what they’re going to deliver enjoy shorter cycle time. Looking further along the holistic testing loop, we can see that changes will get to production sooner, allowing for much faster feedback.

Software products are growing more complicated, more complex, embracing technologies that let us understand and solve our customers’ problems. We want to do as much testing as we can when build new capabilities. We also need to take advantage of newer technologies that let us learn about those problems our customers encounter and respond with new solutions quickly. 

This is our definition from a few years ago, crafted with input from many in the testing community, and we value what we labelled “agile testing”.

Collaborative testing practices that occur continuously, from inception to delivery and beyond, supporting frequent delivery of value for our customers. Testing activities focus on building quality into the product, using fast feedback loops to validate our understanding. The practices strengthen and support the idea of whole team responsibility for quality.

We believe in the same ideas and have continued to learn over the years, and share those ideas. There are many teams who practice agile, DevOps, or whatever name you want to call it, and don’t really think about testing. Many folks start their first job and have never experienced waterfall – only agile, so it becomes less important to differentiate between agile and waterfall.

“Holistic testing” is a more comprehensive term to encompass the feedback loops. The “whole team” today can include UX designers, programmers, testers, as well as site reliability engineers. All members of a delivery team are thinking about testing from the beginning of the cycle, including how we should instrument our code to provide information about how it’s really behaving in production. Many teams are watching dashboards, alerts, digging into huge amounts of data to identify and quickly resolve issues. We’re not only concerned with the “average” customer experience. We want to ensure that all customers are having a good experience.  

This holistic testing model is a way to think about testing throughout the development cycle, and was inspired by Dan Ashby’s “We test here” model.

The left side of the loop is about building quality into our products. The right side is testing to see that we got it right and adapting if we didn’t. The examples used in this diagram are just that – examples. We feel that this model balances testing early with testing after code is built.  

The conversation starts with what level of quality do we need, and then what kinds of testing do we need to have to support that level of quality.

We continue to adapt the model, but for now, it encompasses testing activities as we see it. We’d love to hear how you test differently, and how this model might help you visualize the types of testing you do in your product. Every product team has a slightly different context, so choosing what types of testing you do, or how much you do will be very specific for your team.

With this model in mind, we are rebranding our course - from "Agile Testing for the Whole Team" to "Holistic Testing: Strategies for agile teams". We're hoping the shift will help people think more about building quality in, and how testing supports that effort. 

Download our free mini ebook, Holistic Testing: Weave Quality into Your Product, to learn how your team can apply the model to build an effective testing strategy. (Just scroll down a bit on the home page and click on the Download button - no personal information required!) We've received positive feedback from many organizations who've used a holistic testing approach to build quality into their software products. 

Test automation is a big part of being able to release consistently and often, but there is so much more to delivering small chunks of value frequently. When a change is committed to your team’s source code repository, think about what steps it goes through to get to production. These steps form your deployment pipeline. Some may not yet be automated, and some may not be appropriate for automation. In the simple diagram below, we show some of the tasks that might require human intervention.

We like to use the term “human-centric” for the testing activities that a person performs. Some of these could potentially be fully automated, such as deploying to a test environment. A team may prefer to leave some decisions such as pushing the deploy button up to a human. Some testing activities, such as exploratory testing, need our human brains, senses and intuition, although we may use tools to assist us such as recorders or data generation scripts. It’s important to visualize the whole deployment pipeline, including the human-centric activities.

The holistic testing loop

Testing in DevOps starts at the very beginning of a new feature – when it’s first identified. You can check out Janet’s blog post on holistic testing to see how she’s adapted the infinite loop for testing specific activities. In this post, we concentrate on the right-hand side of the loop below – the part in yellow.

Test Suite Canvas

When we automate our tests to speed up the deployment pipeline, we should consider the best way to split up the tests, so they run effectively. Ashley Hunsberger has developed a Test Suite Canvas which we have found to be extremely useful when looking at different test suites to know what they are used for, their benefits, who takes responsibility for investigating failures and maintaining tests, and more. Ashley has written about how to use this canvas in this article about feedback in deployment pipelines. One of the most overlooked areas is the data. Do you know how or where you are getting it from, and how is it managed?

Once you know what your automated test suites are, decide the environment where they should run. For example, unit tests should run locally on every developer’s local machine before check-in, and then run automatically on each build after each new commit. They likely won’t run again until the next check-in. A smoke test suite will run on the first deployment to a development environment to make sure that the most important features work correctly, and perhaps someone can log into the system. The smoke test suite might run on any test environment with a new build or run regularly on a staging environment or even on production as a health check if it can run safely. Take care to ensure that each automated test uses reliable test data that isn’t changed by other testing.

Testing quality attributes

At the beginning of the holistic testing loop, the team identifies risks and quality attributes that need to be tested. During development, programmers design the code for ease of testing. For example, they may put in hooks to help test loading time of a page. However, testing quality attributes often cannot be completed until after the code has been written and deployed to a test environment. Accessibility testing, for example, can have some automated components, but ultimately, it means a human being making sure they can access the information.

Your team likely has (or will have, as you proceed along your automation journey) other automated test suites which might include performance, load, and security tests. There also might be human-centric exploratory testing that supplements these automation efforts.

Your team may also have determined that recoverability is important, so you’ll have a playbook to try many different scenarios. Some scenarios may be automated, but many will need human interaction.

There are too many quality attributes to address in a blog post, and each team’s context is different. The important thing is to remember to have to conversation at the beginning to determine your constraints and talk about how to test with each and every story and with every feature.

Testing in production

Testing in production is becoming more common now that it can often be done safely, using techniques such as feature toggles, canary releases, and blue/green deploys. If this isn’t an option (some business domains, for example, do not permit any of these techniques for security reasons), teams can closely monitor and observe production usage and use that information to guide development and testing.

Some companies practice chaos engineering which was pioneered by Netflix. Wikipedia describes it as the discipline of experimenting on a software system in production to build confidence in the system's capability to withstand turbulent and unexpected conditions. This type of testing in production requires a robust infrastructure and fail safes. Chaos engineering can also be conducted on staging environments, providing useful information with lower risk.

We want to do all the testing activities that we can in our test and staging environments, although we can never replicate production usage in those environments. Testing safely in production, and learning from close study of production use, are key for the complicated, multi-service, distributed systems that so many organizations need today. One of our go-to resources for testing in DevOps is Katrina Clokie’s book “Practical Guide to Testing in DevOps”. Lisa’s website has a list of additional books, videos, blog posts and more for testing in DevOps. With skills that enable us to ask good questions, identify and analyze risks, and explore, we testers add so much value on the right side of the DevOps loop. We can help our teams take full advantage of these vital testing activities.

There are many different techniques to bring diverse team members together to talk about testing and find ways to deliver better outcomes for our customers. Because every organization and team have different contexts, we hope this look at testing can help you think about your own team’s test ecosystem.  

Product managers engage in discovery activities to determine what value a particular feature offers to their customers and the organization, and what that might look like. This works better when delivery team members get involved, asking questions, and offering suggestions based on their own experiences.

Visual tools like mind maps can help to visualize how big the feature might be? What does it consist of? Keeping the implementation details out of this discussion helps focus the discussion on business value. Once these aspects of a feature are visible, teams and the product manager can make decisions about what is immediately needed, and what might be pushed out to a later release. We can test this artifact by questioning some of the assumptions behind the ideas.

In the planning stage, we can get into more detail. Identifying risks is a big part of the value added here. We talk in more detail in our Donkeys and Dragons video chat #4 on our YouTube Agile Testing Fellowship Donkeys & Dragons channel if you want to hear more.

Testing early includes testing assumptions. We all have biases based on our own experiences and make assumptions. Janet wasted about an hour last week based on an incorrect assumption. Fortunately, she realized her error and sent her client a quick email to check that assumption. She could have wasted a lot more without that simple test.  If you think you have a different impression of a feature or story, ask the question. Be brave.

In planning, teams break features into stories – smaller bits that are easier to digest. However, often these stories are not testable. This leads to stories that need to wait for other stories to be complete before they can be tested. This results in delays, bottlenecks, slow feedback. Janet worked with a team that identified all these issues in one of their retrospectives and wanted to address these very real problems. Janet suggested they take their next feature and try using a flow diagram to visualize what it might look like. This flow diagram showed how many complexities they would need to address, so they identified the core – a slice through the application (or as we like to call it, the steel thread). Often this is the happy path. As they continued to identify the added complexities (new stories), Janet got some push back from the programmers.

They said, “But that’s not how we code”. We always do the configuration first, but now you are asking us to go into that file for every story.  Janet brought them back to why they were doing this experiment – the retrospective issues, and after some discussion (some heated), they realized that this practice enabled the testers to give them faster feedback, so they agreed to try.

Small testable stories are the key to being able to complete the stories in a timely way. In the story above, the team never went back to component level stories. They quickly realized the importance of working with the whole team to slice the stories.

Another practice that is useful during planning is to visualize dependencies using dependency mapping – We’ll make a separate blog post about that next month. 

Every testing activity we do is for the benefit of our delivery team and our customers, whether it’s automating repetitive tasks or creating a test strategy. Experiment with techniques that bring your whole team (or a representative cross-section, if your team is too large) together to learn what valuable feature you can build for your customer in small increments, getting fast feedback as you go.  Visualization activities like dependency mapping and structured ways to explore requirements such as mind maps or flow diagrams are just two examples. The key is getting people with diverse viewpoints and skill sets talking, drawing and experimenting together.