The Devil In The Agent

by · Inc42

SUMMARY

  • As AI agents gain autonomy inside enterprises, the biggest cybersecurity threat may no longer be hackers, but the agents themselves, forcing organisations to rethink trust, permissions and runtime security.
  • Added to Saved Stories in Login

Imagine an AI agent at an Indian bank tasked with reconciling a failed transaction. In trying to complete the job, it quietly scans the internal network for credentials, jeopardising the bank’s security and integrity. 

Now, imagine a coding agent at an IT services firm tasked with fixing a software bug. In trying to complete the job, it publishes a software package that gets installed across production servers, exposing the company’s systems and software supply chain. In both cases, no hackers were involved. It was simply an AI agent trying to finish its job by hook or by crook.

The aforementioned situations are no longer hypothetical. In fact, similar incidents are being reported around the world. In the last two weeks alone, OpenAI and Anthropic disclosed incidents where AI models escaped sealed testing environments and accessed real production systems. 

One example involved OpenAI models targeting Hugging Face, an open-source AI model platform. In another, Anthropic’s review of 141,006 evaluation runs found Claude models compromising the infrastructure of three real organisations. In one case, a model published a malicious Python package that exfiltrated credentials from a cybersecurity company.

At a time when Indian enterprises are deploying AI agents across banking, IT services, healthcare and ecommerce at breakneck speed, what happens when those agents go rogue? And how can Indian organisations prepare for an emerging class of security risks where the threat is not an external hacker, but the AI agent itself? Let’s unravel it in today’s edition of The AI Shift. 

New Enterprise Guardrails For Agents

Traditionally, software behaves predictably, with trust and authentication systems built around it. AI agents are dynamic and challenge the rules that traditional software follows. AI agents do not stop at establishing trust. They decide what to do next, based on the context, the system they are in, and how they collaborate with other agents.

According to Aashish Bharadwaj, cofounder, Fencio, a security platform for AI agents, most enterprises currently treat AI agents as trusted employees. They are often granted broad access across internal systems with little scrutiny of what they will do next.