Why AI agents in financial services need a job appraisal, not just a benchmark 

By Haris Pylarinos, Founder and CEO, Hack The Box 

The financial services industry understands competence can’t be assumed. People working in critical roles are recruited against defined requirements, trained, supervised and assessed. Regulation also sets expectations for managing risk and operational resilience, including testing important business services, identifying vulnerabilities and addressing weaknesses. It is important to understand where risk lies and ensure that responsibility is proportionate to capability. 

AI agents are starting to do work that has consequences for customers, operations and security. For example, they may help investigate suspicious activity, support fraud analysis, summarise alerts, assist with compliance tasks or automate parts of an operational workflow. However, the evidence used to approve an agent is often answering the wrong question. A successful demonstration or strong benchmark result may show what a system is capable of under test conditions, but it doesn’t show whether it can perform the specific role an organisation intends to give it.  

Demonstrations and benchmarks may be useful indicators, but they don’t tell you whether an AI agent can consistently and reliably perform the specific job it is being given within a financial services environment and an organisation’s own controls. 

From benchmark scores to job-role competence 

A model benchmark measures performance against a defined set of tests, but it does not show whether an AI agent can operate securely in a financial services environment. An agent tasked with investigating suspicious transactions, analysing security alerts or supporting incident response must work with specific tools, permissions and sensitive data. It also needs to recognise threats, handle ambiguous evidence and know when to escalate. In financial services, mistakes can have serious consequences for security, compliance and business continuity. This means assessing competence requires testing how agents perform in realistic, role-specific scenarios, not just how well their underlying models score. 

We need to move from treating AI assessment as a one-off technical exercise to giving agents ongoing appraisals of their competence. Before an agent is given responsibility, organisations should define the role it is expected to perform and the standard it must meet. They should test it against realistic scenarios and assess not only whether it reaches the right outcome but whether it follows the right process, stays within its authority and recognises when human intervention is required. 

An agent that helps triage security alerts should be evaluated on whether it identifies relevant signals, handles uncertainty appropriately, follows policy and stays within its permissions. A useful assessment should also show where it fails, what information it overlooks and whether its performance is consistent across different scenarios. The same principle applies wherever AI is used. Tests should reflect the actual responsibilities and consequences associated with each use case. 

An AI agent should not receive a permanent licence to operate based on a single successful assessment. Models are updated, tools and integrations change, workflows evolve, and attackers adapt. An agent that performed well during a pilot could behave very differently months, or even weeks later. Assessment has to be repeated, with clear thresholds for when an agent can act independently, when it must seek approval and when a human must take over. 

Balancing autonomy with accountability 

That does not mean every action needs human sign-off. Excessive oversight can undermine the value of automation. The goal is to match autonomy to demonstrated competence and potential impact. Low-risk, reversible tasks may allow greater independence, whereas decisions that could affect customers, expose sensitive information or disrupt critical services may require stronger checks, tighter permissions and explicit human approval. 

There is also an important human dimension to AI readiness. The people directing agents need the judgement to know when to trust an output, when to challenge it and when to intervene. They need enough domain expertise to recognise when an apparently convincing answer is wrong, understand the consequences of acting on it and take control when the agent moves outside its expected boundaries. Human oversight only works when people have the skills, context and authority to exercise it. 

Alongside model risk assessments and compliance checks, financial institutions need evidence of operational competence: what an agent can do, under which conditions, with what safeguards and how that capability changes over time. This evidence should inform deployment decisions. 

Achieving this requires realistic environments in which to evaluate AI agents performing defined tasks, alongside assessments of the judgement of professionals working with AI. The underlying principle is that readiness should be demonstrated through performance and not inferred. 

As agentic AI moves into increasingly important financial services workflows, organisations need to establish whether an agent is competent for the job, not just whether it can complete a task. That competence must be demonstrated, including how the agent performs when conditions change. The financial services industry has long understood the importance of evidence, controls and accountability. The same principles must be applied to the use of AI. AI agents must be held to a standard that reflects the actual responsibilities they are being given. 

spot_img
spot_img

Subscribe to our Newsletter