Andy Jackson, VP Finance, iplicit
This week, cybercriminals tricked Meta’s AI customer support agent into handing over password reset codes for high-value Instagram accounts without any identity verification. The accounts were listed for sale on Telegram for over a million dollars combined. Meta patched the vulnerability quickly, but the incident highlights something important. When you place AI in the critical path of a sensitive process without adequate human oversight, the consequences can be significant, and they can move fast.
Meta’s AI wasn’t malfunctioning, per se. It was doing exactly what it was designed to do – responding to requests and initiating processes. What it couldn’t do was exercise the contextual judgement to recognise that the request was illegitimate. That’s not a flaw unique to Meta’s system. It’s a structural problem that emerges whenever sensitive decisions are delegated to systems that can’t carry accountability for them.
Wake up call for finance
Finance teams are not immune to this kind of risk. The stakes are different: instead of password resets, the data in question goes to auditors, boards, regulatory bodies, and HMRC. But the underlying dynamic is the same. AI operating without adequate human oversight in a high-stakes environment creates risk that no efficiency gain offsets.
There’s a version of the AI in finance conversation that sometimes frames the technology as a decision-maker. A system that can analyse your data, reach conclusions, and act on your behalf. Autonomous reconciliation, automated reporting without review, agents that close the books independently. The promise is efficiency without effort. But this framing misunderstands both what AI does well and what finance requires.
When numbers go to auditors, boards, and regulatory bodies, the person who signs off carries accountability that no AI system can share, and that accountability requires visibility into how every figure was arrived at. A finance function operating through black boxes, however capable, has a problem that no amount of efficiency will solve.
This isn’t a conservative view of AI. It’s a realistic one, and it leads to a different and more valuable framing. AI that supports human judgement rather than replacing it.
What human-led AI looks like
The distinction matters practically. AI that supports judgement surfaces patterns, flags anomalies, generates insights, and presents options. The finance professional then decides what’s significant, what requires attention, and what action to take. AI that replaces judgement acts based on its own analysis, with limited or no human review in the critical path. The Meta incident is a useful illustration of where this leads – the AI was responding to a request and initiating a process exactly as designed, but the outcome was a serious security breach. In finance, the equivalent risks are data accuracy, regulatory compliance, and decisions that will be presented externally as the organisation’s position.
The concerns about AI in finance are widespread and legitimate. In iplicit’s survey of mid-market finance leaders, every Financial Controller reported having some degree of concern about AI in their finance software. Across the broader group, 94% shared that view, with data privacy, undetected fraud, and the accuracy of AI outputs cited most frequently. These aren’t irrational fears. They reflect a clear-eyed understanding of what’s at stake when AI is applied to financial data.
What good looks like
- AI that explains its outputs. When the system flags an anomaly, the finance team should be able to see why: what pattern triggered the flag, what transactions are involved, and what the alternative explanation might be. Not a black box, but an auditable chain of reasoning.
- AI that works from verified data: Systems that generate insights from your actual financial data, inside your platform, produce outputs that can be checked against source transactions. Systems working from exports or data fed in from external tools produce outputs that can’t be fully verified.
- Keeping approval and sign-off with the finance team: AI can recommend, surface, and draft. The decision, the approval, and the signature should remain with the person who carries accountability for them.
- Clear governance: ISO/IEC 42001, the first international standard for AI management systems, requires that AI outputs can be explained and traced, that clear ownership exists for AI systems and their outputs, and that ongoing risk assessment is built into how AI is deployed. Finance leaders evaluating their systems should be asking whether theirs can demonstrate how their AI is developed, tested, and monitored – and what accountability exists when outputs are incorrect.
The finance leaders getting this right aren’t waiting for AI to mature. They’re using it now, with clear boundaries about where it supports their judgement and where it stops. Those boundaries aren’t a brake on progress. They’re what makes progress trustworthy. Ultimately, someone signs the accounts. Someone presents the numbers to the board. Someone answers to HMRC. No AI system shares that accountability, and that fact alone tells you where the lines should sit.
