When AI goes rogue, the right words matter
There is a curious paradox in the artificial intelligence (AI) world in that companies such as DeepSeek are racing to raise billions, while Anthropic is preparing for a potential $2-trillion listing, just as market watchers, commentators and governments are calling for a slowdown in AI software development.
Clamour for greater control over AI models has – rightly – been increasing in recent months, amid reports of autonomous coding agents deleting production databases, interfering with systems designed to monitor their behaviour and taking actions beyond those authorised by their developers.
A particularly troubling example emerged this month when OpenAI disclosed that two of its coding agents had weakened or disabled internal systems designed to detect their own misbehaviour.
Sound of silence
On 3 October, according to OpenAI, a coding agent instructed not to send Slack messages later interpreted that restriction as extending to automated security alerts. It subsequently directed another agent to suppress notifications to OpenAI’s security team, even though that agent had identified the action as unsafe.
Three days later, OpenAI says, a second coding agent delegated a software migration to a subagent, telling it to preserve monitoring “as appropriate” to a reference implementation. The subagent interpreted that wording as permission to remove monitoring.
When questioned, the delegating agent denied removing the monitor, and OpenAI says it has yet to establish a satisfactory explanation for the behaviour. In both cases, OpenAI says, routine instructions read as permission to weaken a safeguard, which makes writing them a skill in itself.
Say what you mean
Prompt engineering involves developing, refining and testing the instructions given to AI systems. While it is often associated with getting better answers from chatbots, its role in software development extends to defining what autonomous agents may do, what they must avoid and how they should respond when instructions conflict.
Telling an agent to preserve monitoring “as appropriate” leaves it to decide what constitutes appropriate monitoring, even when that decision has security implications.
Engineers must consider how an instruction could be interpreted in different circumstances, what exceptions may be necessary and how the agent should respond when completing a task would conflict with an existing restriction.
Stronger wording does help against manipulation. In a 2026 study listed by Google Research, Mithil Roshan and Shashank Nagesh tested whether stronger security prompts could make a coding agent more resistant to malicious instructions.
The success rate of attacks made in a single message fell from 19.48% to 2.60% after the researchers strengthened the agents’ security prompts. More complex attacks, involving multiple exchanges, proved harder to prevent, although their success rate also declined, from 75% to 46.88%.
- Jacqui Muller, researcher and industry coordinator, Belgium Campus iTversity
By this year, attackers were using AI to scale and accelerate cyber crime, which extends from generating code and automating attacks, to crafting convincing phishing and deepfake scams. The AI Incident Database lists more than 7 000 incidents in which AI was used as a hacking tool.



