Over the years Janet has done many talks and keynotes about quality, and what it means – from a process perspective as well as a product perspective. Often, teams and organizations think about quality with a very narrow view.
Jerry Weinberg described quality as “value to some person”. While this is true, it doesn’t help us evaluate our product quality to be able to release it to our customers with confidence.
5 quality approaches – David A. Garvin
Janet was introduced by Isabel Evans to the idea of five different approaches to quality from David A. Garvin (1984). Looking at the different approaches made us realize that perhaps, we can expand our views and look at quality differently. The five approaches to quality he talks about are:
- Manufacturing-based – think practices to supply / develop the product
- Product-based – it’s about the features, the quality of the attributes
- User-based – this aligns with Jerry Weinberg’s definition, is the user satisfied?
- Value-based – defined in terms of cost and price; think trust or safety
- Transcendent – universally recognized, but hard to quantify; it’s about the emotion a product evokes
Garvin's 5 quality approaches
Not every product we build needs to satisfy each one of these approaches, but if we start to think about quality a bit differently, we can learn to measure it better. We will start to understand what is “good enough”.
Contemporary quality engineering – Anne-Marie Charrett
Another viewpoint of quality has been supplied by Anne-Marie Charrett She talks about “contemporary quality engineering”, by which she means building quality into our products, rather than an older view of “quality engineering” which was adherence to a specific process.
Anne-Marie helped us start looking at quality from a different perspective by coining a new term: Qualitability. She uses it to describe how easy or hard is it to know (and see) the state of quality of your product at any given time. One way to gauge the state of quality is by using metrics. It can be difficult to come up with metrics that really measure whether your team is achieving its quality goals. Anne-Marie advised that the delivery team should choose the metrics, based on business outcomes and on what they want to learn about their product. Make quality visible.
Emergent Quality diagram by Anne-Marie Charrett
Quality metrics
In one team Lisa worked in, they chose metrics based on how new customers used their product (a test automation tool). For example, an effective automated regression test needs to have assertions, so they looked at the average number of assertions customers put into their tests. If tests lack assertions, it may mean that the way to create them is not obvious enough to people using the tool. That’s reducing the level of quality for everyone. This metric is probably not one than many other teams would use, but it worked for their context.
We may need to rethink the list of quality attributes we use. For example:
Delivery frequency: How do we break features down into smaller pieces, so that we can get small changes to the customer more often without negatively impacting the business.
Value: Is what we deliver valuable to the customer?
Feedback: Does our process give us quick feedback on the quality and value of changes?
Recoverability: This is not new, but for many contexts it changes its meaning from “Can we recover to a known state” to “How can we get a fix out very quickly without impacting the business”. Feature toggles, blue-green deployment, robust and fast deploy pipelines all contribute working towards this attribute.
Quality expectations
When Lisa’s team experienced more regression failures in production than they wanted, they brought in a tester on a consultant basis who spent several weeks improving the automated UI regression test suite. At the end of his engagement, he gave a terrific presentation to everyone in the company that included the most critical bugs he’d found, as well as his ideas for new features that would improve the product. That immediate visibility led to the team implementing some of the features quickly.
Expectations of quality change as you experience different levels of quality. Like buying a new car, if you never had it, you won’t miss it. For example, when Janet bought a new Mini-Cooper, it came with a heads-up display, and now she wouldn’t think of buying a car without it. If we look at Lisa’s story, although her teammates had suggested some of the same features as the consultant, hearing those ideas from a new perspective, from someone who was also a target customer of the product, was more compelling.
Software testers are often viewed as a safety net, the last line of defense to spot quality issues before new features are deployed to production. For a long time, the focus was on trying to find all the bugs after coding was complete. With agile development, focus shifted to bug prevention, but we still try to detect bugs before release. Now that many teams deploy new changes to production so frequently, it becomes more important to find ways to speedily detect production issues and deploy fixes within a matter of minutes. Old challenges don’t go away, and we have new ones to embrace.
Ask the hard questions
We really like Anne-Marie’s metaphor that “Software testing is the headlights, quality is the journey, and business outcomes are the destination.” We need to tie what we build into business outcomes, and that starts by thinking quality first. Ask the hard questions at the very beginning of the cycle to be sure we are building the right thing, and it is solving a business problem. Also recognizing that quality isn’t only about the product quality - we also need to consider everything that goes into making our product - the pipeline, processes, practices and the people.
There are many different ways of looking at quality, and the discussion is becoming more mainstream as teams and organizations struggle with what it means to them and their products.