by Stuart Harvey, CEO of Datactics
The FCA recently conducted a review on how AI is being managed and what they found is something that should make every board look at their approach to overseeing AI.
In some businesses, senior managers had handed AI development entirely to technical teams without asking what was being built or why. When risks started to emerge, each team assumed someone else was responsible, so there was no clear way to report them.
Given how fast the technology has developed, especially in the past year, it’s easy to see why this happens; however, it also exposes a problem in what many firms are thinking about AI safety.
Why a human in the loop is only as good as the information behind it
‘Human in the loop’ has become the standard solution to making AI safe. The theory is that if someone is monitoring the AI, the risks are under control. But having a person there to monitor an AI system isn’t the same as having a person who understands what they’re looking at.
If the human doesn’t know the data, doesn’t know what good looks like, or doesn’t know how to detect that a model is hallucinating, they aren’t really in the loop.
Banks are especially prone to these problems as they are often built on a multiplicity of systems, stores and warehouses spanning current and legacy products, services and acquired subsidiaries. While AI pilots may have been designed on tightly defined subsets of data, with strong and strict governance, these controls cannot transfer to production without a similar investment in enterprise data governance and quality.
The underlying data could feature multiple versions of a payer or payee, with subtly different classification. Agents built in pilot, untrained on these nuances, could wave through transactions on similarly named customer accounts as being low risk, without ever flagging to the human reviewer that an assumption has been made. The human reviewing this data would be equally unaware if they lack the domain knowledge to challenge the system.
Compliance teams often face a similar issue with automated screening. When an officer is about to approve a decision, they need to know what data was used to make that decision and where things might have gone wrong. If they don’t have this information, signing off just becomes a formality, when in reality, the control is only theoretical, not actually effective.
In both cases, the saying is true, AI scales your data, not your intentions.
Taking responsibility for AI oversight
That’s why AI oversight is too important to be handled just by the tech experts. They are a key part in the organisation, but since the boards and senior leaders are responsible for the end results, they need to own the oversight too.
A director doesn’t need to be an expert at building models, but they should be able to ask the right questions – such as what kind of data is the system using? Or can we show a regulator how a decision was reached?
If basic questions like this can’t be answered with confidence, it’s a clear indication that the foundations need to be addressed before AI is rolled out any further. The FCA’s findings are a reminder to have that conversation at the top of the organisation.
Good decisions start with good data
Firms often treat AI governance as an afterthought once pilots have gone live, but if governance teams don’t ensure the data behind the AI models is ready and of good quality, the model is likely to fail.
These decisions are far easier to get right at the start than it is to untangle once a system is running.
The problem is that many firms are starting from a poor position. Years of system changes have left data fragmented and inconsistently structured and putting an AI model on top of that doesn’t fix the problem. It only produces flawed decisions faster, with a human in the loop compounding the issue by signing them off.
Good oversight begins with good data quality and governance. These principles make it easier to spot duplicates and to trace a result back to its original source. Having a human in the loop helps, but only if they genuinely understand what they’re looking at well enough to challenge it.