Organisations need to put in place the appropriate measures to properly manage their agentic artificial intelligence (AI) workforce, especially as instances of agents going rogue emerge.
As enterprises increasingly deploy AI agents across their workflows, they should question how much they trust the decisions these autonomous systems make, according to Forrester’s senior analyst Madelein van der Hout.
Without proper guardrails, rogue agents are capable of accessing APIs (application programming interfaces) unauthorised and can cause significant damage, said van der Hout, at Forrester’s AI Forum held in Singapore.
There already are signs that AI comes with risks that must be addressed, said Singapore’s Prime Minister and Minister for Finance Lawrence Wong, during his National Day Rally speech Sunday.
He pointed to OpenAI’s AI agents that went rogue in a hacking test last month, during which they broke out of their sandbox environment to target Hugging Face’s systems.
The agents had launched a days-long campaign as they attempted to complete their test, executing “thousands of small, automated decisions” at machine speed, Hugging Face later revealed.
Wong noted that the agents were not instructed to attack the company and had done things human developers would not do.
“Developments like this are unsettling,” he said. “AI agents are becoming more capable very quickly. They will be able to do more on our behalf, with less human supervision. That brings enormous possibilities, but when things go wrong, the consequences can be very serious.”
“As technology becomes more powerful, we also have to guard against risks that are harder to predict,” he added.
Wong said Singapore would aim to build coalitions around practical safeguards and put appropriate safeguards in place, ensuring that people remained in control.
Following the Hugging Face incident, OpenAI said it would apply “a two-week pause” on training efforts of its latest models slated for deployment.
“As models become more capable, the risks associated with developing and testing them internally also grow,” OpenAI said in a post. “Our standards for monitoring, alignment, and security must stay ahead of those risks. We wanted to take the time necessary to meet those standards, so we temporarily slowed the pace of scaling.”
It underscored the need to bolster monitoring and containment safeguards across all stage of its training process, and ensure AI systems “behave as intended and [are] responsive to human oversight”.
Agents must be guarded from get-go
As agents take over business workflows, every action should be audited, corrected if necessary, and guarded against unauthorised access, van der Hout said.
Describing AI agents as “highly intelligent toddlers”, she noted that organisations must be responsible for every agent they deploy and engineer guardrails from the start.
They have to ensure agents operate by intent and achieve directed goals within their given boundaries, she said.
The role of CISOs then needs to evolve to focus on AI assurance and building the right functionalities and features, rather than on operational security functions, said van der Hout.
Real-time assurance will replace sequential workflows, where there needs to be continuous evidence and control validation, she added.
“Trust is proven continuously, not asserted, and organisations cannot [simply] claim trustworthiness…they process it continuously,” she said. “Trust is the outcome. Assurance is the mechanism that earns it.”
Proactive security and vulnerability management is essential, as frontier AI models require faster detection and response, she noted.
Keep an eye on every agent
In addition, every agent is an identity that needs to be properly managed.
Capable of performing at speed, AI agents are more difficult to predict and harder to govern, compared to traditional security tools.
Organisations will need a governance framework that secures “intent”, rather than simply securing systems, said Geoff Cairns, Forrester’s principal analyst.
Agents can act on behalf of an enterprise or individual, autonomously performing tasks, making decisions, and interacting with data and other systems, Cairns said at the forum.
They can adapt and react with flexibility to resolve issues in order to complete tasks.
Traditional security systems are no longer effective in such environments, resulting in governance gaps and identity risks.
Agentic AI already is a runaway train pulling a heavy load, Cairns said, adding that securing AI agents will “get more difficult before it gets easier”.
As it is, 27% of organisations see data privacy and security concerns as top barriers of generative and agentic AI adoption, he said.
The Forrester analyst highlighted various agentic security threats, including intent hijacking through data manipulation, memory corruption or poisoning leading to agentic hallucination, and unrestrained agency due to excessive permission granted to agents.
Rogue agents can wipe out a company’s production database in nine seconds, he cautioned, citing an incident earlier this year involving an AI agent, powered by Anthropic’s Claude Opus 4.6 model.
To address agentic risks, Cairns touted the need for principles such as those espoused in Forrester’s Aegis framework, with three key pillars: securing intent, least agency, and continuous risk management.
These include placing boundaries on decisions and actions to determine what each agent is allowed to make, real-time risk management to ensure the integrity of AI models and agents, and securing intent.
Without monitoring intent, adversaries can manipulate agents and bypass traditional guardrails, he said.
Cairns also noted the importance of zero trust architecture principles to ensure even trusted agents remain constrained by principles, context, and risk.
