By Philip Dutton, Founder & President, Solidatus
The term ‘automated discrimination’ sounds like something that could be dreamt up by a Hollywood screenwriter. The sort of phrase attached to a dystopian future where an all-powerful algorithm decides who gets a mortgage, who receives a loan and who is quietly locked out of the financial system. Unfortunately, it’s becoming much closer to reality than many banks would like to admit.
Artificial intelligence, now embedded across financial services, is being used to combat fraud, overhaul customer service, streamline compliance processes and support lending decisions. Done well, AI can do these things much more efficiently, drive genuine positive changes in the industry and help financial institutions make better decisions. The problem, however, arises from the fact that AI inherits the strengths and weaknesses of the data it consumes. If that data is incomplete, inaccurate or poorly understood, the results can be far more damaging than most organisations realise or are prepared for.
When bad data becomes automated decision-making
Conversations around AI often focus on the sophistication of a model or the size of an investment, but much less attention is paid to whether the underlying data can be trusted. Yet the oldest rule in technology still applies – feed flawed information into a system and you should not be surprised when equally flawed outcomes are found on the other side. It’s the foundational mathematic principle of garbage in, garbage out (GIGO) – a concept coined by mathematician Charles Babbage that predates modern computers by around 200 years.
It’s unlikely though that Mr. Babbage would have predicted just how much garbage could be in the system in the first place, and just how critical the outcome could be. For banks, the stakes are especially high. A poorly governed AI model could, for example, influence thousands of lending decisions before anyone spots a problem and, as AI scales, so does the size of the risk. What was once a data quality issue can become an enterprise-wide problem with unprecedented regulatory, financial and reputational consequences.
The financial industry is also particularly exposed. An overwhelming 93% of the sector recognises that AI will be the main driver for industry change in the next five years, according to Lloyds Bank’s latest Financial Institutions Sentiment Survey. This is not a surprise in and of itself as few, if any, technologies have had the potential to improve productivity, reduce costs and create competitive advantage on quite the same scale.
But gung-ho AI attitudes aside, history suggests banks in particular should have a more cautious approach. Financial institutions have spent more than a decade grappling with the challenge of understanding their own data. Despite BCBS 239 being introduced in 2013 to improve risk data aggregation, many large organisations still struggle to achieve a complete picture of where critical data originates, how it moves through the business and where it is ultimately used. If firms cannot fully explain the journey of the data underpinning a regulatory report, it becomes much harder to explain the reasoning behind an AI-driven decision.
The uncomfortable truth
Most AI models are remarkably efficient mirrors, reflecting whatever is fed into them. If, for instance, customer data is incomplete, AI can force conclusions from an incomplete picture. If errors exist in source systems, those errors can ripple through hundreds of downstream processes without anyone noticing. Left unchecked, bias that was once buried in legacy systems can be repeated at unprecedented scale with speed, consistency and the appearance of objectivity. That is what makes automated discrimination such a uniquely modern risk – the machine looks and sounds impartial, even when the data underneath it is not.
With proper lineage in place, the picture changes completely. When an AI model is used to assess a credit application, the bank can trace every data point feeding that decision back to its source. It can check whether the information is accurate, whether it is complete and whether it is quietly relying on proxies that could introduce bias. That visibility is what allows an institution to catch a problem before it becomes a pattern.
This is why data lineage is no longer a technical afterthought, for compliance officers, but a genuine strategic boardroom priority. In simple terms, data lineage gives the ability to trace information across its entire journey – where it came from, how it has been transformed, who has touched it along the way and where it eventually ends up. Think of it as a map of a bank’s data ecosystem. Without that map, institutions are effectively asking their AI models to make decisions in the dark and hoping for the best.
Staying ahead of regulation
Banks have spent years responding to rules written for a pre-AI world . As such, in this new context, it is a fairly safe bet that new regulations aimed specifically at how AI is built, trained and monitored are coming. The deadline for high-risk AI compliance under the EU AI Act may well have been pushed back, but this offers less time than it might appear.
US regulators are already integrating AI into supervisory examinations. Gartner predicts that by 2028, 50% of organizations will adopt zero-trust data governance as AI-generated content proliferates through enterprise data supply chains.
Institutions that wait to be told what to do will find themselves scrambling to retrofit governance onto systems that were never designed for it. Those that anticipate the direction of travel and build strong data foundations now will be in a far more comfortable position when the rules eventually catch up.
The balancing act
None of this means banks should slow down when it comes to AI – the benefits of staying competitive in this race are real, and the pressure to be seen to innovate is only going to intensify. The point is that speed and safety are not opposing forces, but rather, an institution that understands its data can move faster as a result, because it can trust its own systems, explain its own decisions and deploy new models with confidence rather than crossed fingers.
In the end, the winners will be the institutions that can answer three deceptively simple questions at any given moment. Where did this data come from, how has it changed, and where is it being used? Get that right and AI becomes one of the most powerful tools the industry has ever had. Get it wrong and automated discrimination stops being a phrase from a film script and starts becoming a line item in a regulator’s report.