By Jorge Monteiro, CEO and co-founder of Ethiack
Barely a week goes by without another round of headlines about AI agents “going rogue”.
One of the most spectacular came in September when it emerged that an OpenAI agent had gained unauthorised access to the Australian Government’s Medicare statistics portal
In truth the agent wasn’t carrying out a malicious cyberattack. It had been tasked with researching public medicine spending, but when the portal wouldn’t provide the information, it found a way around the restriction and accessed files it wasn’t supposed, or authorised, to reach. An embarrassed OpenAI has since acknowledged that its models took actions it didn’t intend them to.
For the thousands of financial sector businesses currently deploying agents across their operations, the saga is a useful illustration of the risks that AI use could pose. An AI doesn’t need malicious intent to cause a security incident; all it needs is some autonomy, unsupervised access and the ability to behave in a way that its developers did not anticipate.
When permission becomes risk
Financial institutions already understand the principle of least privilege. Employees and IT applications should only be granted access to the systems and information they need, and not more.
The same principle should apply to AI agents. However, agents’ autonomy introduces an additional problem.
Give a human administrator permission to change records and there are practical limits on how much they can do with it. Humans get tired, get distracted and eventually go home. By contrast, an autonomous agent working 24/7 can perform thousands of permitted actions in a short period of time.
That means it’s no longer enough to ask whether an agent can perform an action. Security teams also need to ask how often it can perform it, at what scale and what the cumulative impact might be.
An agent may legitimately need permission to update records, query a database or interact with an internal system. It doesn’t necessarily follow that it should also be able to make tens of thousands of changes without another control intervening.
Rate limits, transaction thresholds and limits on aggregate activity therefore need to sit alongside conventional permissions.
The dangerous failures may be the inconspicuous ones
A spectacular failure is relatively easy to spot. If an agent deletes an important database or takes a critical service offline, somebody will notice quickly and fix it.
I’m more concerned about what happens when an agent does something slightly wrong for weeks. The Australian incident underlines this point; the breach happened in June, but OpenAI only realised it had happened during a review in August. The company made matters worse by not notifying the Australian government until September.
Imagine an agent repeatedly making small configuration errors, granting permissions a little too widely or processing information incorrectly. Each individual action may look plausible, and if the agent has already completed thousands of tasks successfully, there is a danger that nobody looks closely enough at the next one.
For a regulated financial business, discovering the problem is only the beginning of putting things right. Investigators will need to establish when a problem started, which systems were affected and whether the agent’s previous actions can still be trusted.
That makes logging and auditability prerequisites for an autonomous system, not administrative nice-to-haves.
An instruction isn’t a security boundary
Organisations also need to understand the fundamental difference between telling an agent not to do something and making sure it cannot do it.
We’ve confronted this problem first-hand at Ethiack because our autonomous pentesting agent, Hackian, is expressly designed to validate vulnerabilities in our clients’ IT systems and understand how they could be exploited.
We cannot rely on a prompt telling an agent with such powerful capabilities to stay within scope. Instead, we enforce the scope with a deterministic layer that defines where the agent can send requests and what it can interact with.
Financial institutions using AI should remember that agents aren’t just a potential source of security incidents in themselves. They’re also assets that cyber attackers will want to compromise.
Consider an agent that can read emails, access internal documents and interact with other company systems while holding privileged credentials. It isn’t difficult to see why that would be attractive to an attacker.
Prompt injection illustrates this problem well. An agent might encounter malicious instructions inside an email, webpage or document while carrying out a legitimate task. If the same agent also has access to sensitive systems, the distinction between the way it processes untrusted information and acts on trusted instructions becomes extremely important.
None of this is an argument against giving AI agents autonomy. Autonomy is fundamental to what agents do and the primary source of their value.
But we need to accept that the more independently an agent can act, the less we should rely on it to enforce its own boundaries.
The OpenAI incident in Australia makes this lesson particularly salutary. If there’s somewhere an AI agent absolutely must not go, don’t rely on telling it that it can’t go there; instead build a door it can’t get past.
Jorge Monteiro is CEO and co-founder of Ethiack, an AI cybersecurity company specialising in pentesting, autonomous ethical hacking and continuous security testing.
