AI Accountability

A while ago, one of our readers commented:

“Something that always gets missed or ignored when talking about AI is accountability, it's not really possible to hold an AI accountable? so who takes accountability for the mistakes the AI inevitably makes, or perhaps you do have a way for holding the AI accountable? This should be included in every talk about AI.”

It’s a topic we thought we should delve into a bit more. Our first question to ourselves was:  was who could possibly be held accountable? The list could be very long, and might include:

• The people who trained the AI system
• The management / executive who mandated the use of AI/LLM
• The people who chose which AI/LLM to use
• The development team who uses the AI/LLM to create their own system(s)
• The individual who codes and/or tests the features developed

Neither of us believe in a single “wringable neck” for accountability (or blame). AI-assisted software development is still in its infancy and evolving on almost a daily basis.

Mitigation vs accountability

Let’s work to reduce or mitigate the risk rather than on focusing on who’s to blame (accountability).

Most of us realize that there are many biases built into the different LLMs depending on how they’ve been trained, and what information was used for the training They are known to make up answers to prompts in many situations. Often, the answers are based on faulty information due to the choice materials on which it was trained. LLM systems often focus on different areas of expertise.

If we step back a bit and look at software development, teams have faced many of these issues before. The difference is the pace at which folks are challenged to process  new information and technology.

Good practices

Let’s go back to basics. We know that having one person doing all the work, is irresponsible. It leads to mistakes, misunderstandings, incorrect assumptions and often introduces other errors  that are not often caught until it’s too late. It’s why code reviews and testing were introduced.

There is risk involved in any kind of development. With iterative development, teams learned to shorten feedback cycles by using peer reviews, pair programming, testing early, and having more than one role discuss what was needed. These basic techniques can and should be used when using AI assistance as well.

Consider the Holistic Testing Model and apply the principles to building software with AI tools. Here are a few examples:

1. Understand the pros and cons of the LLM you are using before you start.
2. Understand the problem you are trying to solve before you start.
3. Consider pairing when using AI to create code so there are two pairs of eyes on the prompts and answers.
4. Slice each new change into small, testable stories. Focus on delivering one small story at a time. Peer review the code that is generated – this may be more often than you are used to doing it. The same techniques apply when writing tests.
5. Test the AI infrastructure. For instance, agents can degrade, so be sure to run automated tests on them continually.
6. Learn from your choices, and adapt as you go.