A question we get once in a while (though less often than in the past), is about UAT (user acceptance testing). People wonder how it fit into an agile cadence – especially when the feature spans more than one or two iterations.
User Acceptance Testing (UAT) is important in large customized applications, as well as internal applications. It’s performed by all affected business departments to verify usability of the system and to confirm existing and new (emphasis on new) business functionality of the system. Your customers are the ones who must live with the application, so they need to make sure it works on their system and with their data.
Teams that still perform UAT, likely do not have continuous delivery in place. However, that is not a rule and some clients still want some form of final say before they accept a new release.
In one team that Lisa worked in, prior to a deploy to production, they first installed the release candidate in the pre-production environment. Lisa met with the client’s test manager and explained all the testing already completed – automated tests at the unit and UI level, acceptance testing, and exploratory testing. This was a totally new idea to them, and they were thrilled that they could cut way down on the UAT. Since several different teams also delivered changes to the system, any issue found was reported during a conference call and the appropriate team took charge of it. They had working agreements with the customer about what would constitute a critical show-stopper bug that they had to fix and what could be turned into a user story and put off to the next release.
Janet often tells a story of a team that had three-month release cycles (quarterly). The day she joined, they had just put the release into the customer’s test environment for UAT. She was told it would take six weeks. What? How could that be? It’s half-way through the next release cycle. Crazy. Slowly she started introducing the customer representative to features earlier and earlier, and on the second release cycle, when the team delivered the candidate for UAT, the customer quoted one day to test in her environment. Mission accomplished.
Fitting UAT into the Holistic Testing Model
The key is to think about how to get testing moved as early in the cycle as possible. Consider what are the constraints, and how can you remove or mitigate them. Have conversations with your customers in advance to see how much UAT will be necessary and how any bugs found will be handled.
We have not specified UAT as an example of a test in our Holistic Testing Model. Many teams no longer have the idea of a separate UAT if they are collaborating very closely with their customers. Or, they have a less formal UAT if they are practicing continuous delivery and using a release strategy such as feature flags to hide new changes until they have been tested by their customers.
If we added UAT as an example in the Holistic Testing Model, it would fit into the Deploy stage, but that doesn’t mean you have to wait until right before a production release to have customers test the system.
Embracing today’s leading practices for deploying, monitoring, and observing the system in production can mitigate risk by allowing instant response to production issues. Teams have many options. Collaborate with your customers to see what fits best for your situation.