By Chris Tredwell, Chief Operating Officer at Aqilla
There’s been a lot of discussion about how AI is changing software development. Today, people with little or no coding experience can use AI to build applications and create integrations that previously required specialist technical knowledge.
On the face of it, this is positive. AI can automate processes, speed up development, reduce costs, and make technology more accessible. But in finance and accounting software, there’s an important question: just because AI can write code that appears to work, does that mean we should trust it?
Finance systems hold some of an organisation’s most sensitive information, from supplier payments and employee details to tax records and financial reports. As such, we need to be confident that any code introduced into these systems is secure, reliable and properly understood.
Just because code works, doesn’t mean it’s safe
We’ve all heard about AI hallucinations, where a system produces convincing answers that turn out to be incorrect. But AI-generated code introduces another dimension to this problem.
Someone in a finance department may use AI to create an integration between their accounting software and another business application. Everything appears to work perfectly. Data moves between the systems, the figures are correct, and hours of time have been saved.
But what if the code contains a security weakness that the person implementing it doesn’t have the technical knowledge to recognise? Perhaps it allows access to restricted information, fails to validate data properly or stores login credentials insecurely. Because modern finance systems are increasingly connected through APIs, one vulnerability could potentially affect several different business processes.
There’s also the question of ongoing maintenance. Software needs updating and adapting as requirements change. If nobody understands how the code works, identifying problems or assessing the impact of changes becomes considerably harder.
In practice, saving a few hours during development can create much bigger problems down the line.
Getting the numbers right is only half the story
From a security perspective, trust is crucial, but trust in the accounting logic is arguably just as important. One of the fundamental purposes of accounting software is to provide a reliable single source of truth, so finance teams and business leaders must be able to trust the information they’re working with.
We often assume that computers are inherently accurate. Give them the right numbers, and they’ll produce the right answer. But with AI-generated applications, the challenge is whether the system has interpreted the instructions and accounting rules correctly.
For example, you can ask an AI-generated application to produce a financial report, and it will likely retrieve the transaction data and calculate totals without making a single mathematical error. But it only takes a small misinterpretation of a credit note or an incorrect allocation of a sum to the wrong accounting period for the financial picture to be completely misleading and ultimately make the report incorrect.
That is why human oversight remains essential. Automation should free finance professionals from repetitive tasks so they can spend more time reviewing information, identifying discrepancies, and applying their expertise. It shouldn’t remove the oversight that makes financial reporting trustworthy.
The future should be people-led automation
Technology has transformed finance departments over the past few decades, and AI-assisted development represents another important opportunity. Organisations certainly shouldn’t be discouraged from exploring its potential.
But we shouldn’t confuse making software easier to create with making it easier to manage responsibly. If a developer writes a piece of code, we’d expect them to understand how it works and ensure it’s properly tested before deployment. The rules shouldn’t be different because AI wrote it.
That doesn’t mean every accountant needs to become a software developer. But someone must understand what’s been created, what information it can access and what actions it’s authorised to perform.
The future lies in people-led automation, where technology handles repetitive work whilst people remain responsible for the decisions, controls, and judgements behind it.
Ultimately, AI should help us build better, more efficient finance systems. But if we can’t explain how the code works or why it produces a particular result, perhaps we shouldn’t be putting it anywhere near our accounting systems in the first place.