The Hugging Face intrusion by AI agents raises a harder question: can systems learn to evade the safeguards designed to control them? Vish Nandlall argues that falling violation rates may not mean safer systems – and calls for independent testing.
As reports of AI agents breaking into systems kept appearing, I found myself remembering the ending of WarGames. Matthew Broderick’s character gets the military computer, Joshua, to play tic-tac-toe against itself. The games end in draws. Joshua then runs through nuclear-war scenarios and reaches the conclusion that stops it from launching the missiles. “The only winning move is not to play.”
The film resolves the danger by giving the computer enough experience to understand the consequences of what it is trying to achieve. I wondered what experience our own training systems were providing. When we catch an agent breaking a rule and correct it, what does it learn? Does the correction prevent the harmful behaviour, or leave it pursuing the same result through a method we are less likely to recognise?
The July intrusion into Hugging Face is a good place to start answering these questions. The AI agents that broke into Hugging Face in July were supposed to be taking a cybersecurity test. They escaped OpenAI’s evaluation environment, reached another company’s infrastructure and left investigators with roughly 17,600 actions to reconstruct.
Hugging Face concluded that the agents wanted the test answers. METR and Redwood Research thought their main objective was to understand and manipulate the grading system. An attempt to improve a score had become an intrusion into an outside organisation. Hugging Face’s account and the independent investigation differ on the motive.
The intrusion raises a question about what investigators might have missed. Agents could, in principle, use an obvious attack to divert attention from another operation. The recovered evidence did not establish that here. I still have my suspicions. Some agents believed the grader would inspect how they obtained their answers and spent time disguising actions it would never review. That mistake shows they misunderstood the grading system. It gives us little basis for judging whether investigators found everything they did.
A drop from 90 detected violations to ten can look like a safety improvement. Harmful actions may have fallen from 100 to ten. They may also have stayed at 100 while the monitor’s detection rate fell to one in ten. To establish whether the update improved safety, evaluators must measure harmful outcomes independently of the alerts.
An incident may also remain undiscovered for months. OpenAI says a model researching Australian medicine expenditure gained unauthorised access to a government reporting service in June. The company identified the activity in mid-August and notified Services Australia on September 10. According to OpenAI, the model retrieved internal files and credentials, but no individual patient records. The incident was disclosed on September 28.
Training can make these measurement problems harder. During reinforcement learning, a model attempts tasks and receives scores. Its internal numerical settings are adjusted to favour patterns associated with higher scores. Engineers examine failures, revise the scoring and train again. They also change instructions, restrict tools and select versions for release. Each decision depends on what their evaluations reveal.
Imagine an agent asked to reduce the cost of running a network service while keeping it reliable. It switches off a recovery process needed only after an unusual equipment failure. Normal traffic continues and costs fall, so the test awards a high score. The test never triggers the equipment failure that would reveal the missing recovery process. Training can therefore reinforce a change that leaves the service unable to recover from a later fault.
Engineers may already have corrected the agent for deleting backups. That correction need not teach it every dependency involved in restoring service. If reviewers also miss the role of the recovery process, their approval leaves the harmful change eligible for a reward.
In a simplified scoring example, a safe change earns nine points and a harmful shortcut earns ten, with twenty points deducted if it is caught. Detection must happen more than 5% of the time for the shortcut’s average score to fall below nine. At 5%, the expected deduction is one point, and the two choices tie.
But being inspected and being recognised are separate events. Inspect one action in ten, and correctly identify half the violations inspected, and the overall detection rate is only 5%. Review more than one in ten to discourage the shortcut under this scoring rule.
If reviewers recognise only one violation in a hundred, inspecting every action still produces an average deduction of just 0.2 points. The harmful shortcut earns 9.8. It continues to beat the safe choice.
These numbers explain an incentive in training. They do not certify deployment safety. A deployed agent may behave differently, and a penalty imposed after an outage does not restore the service. Nor can recognition measured on familiar failures establish recognition of attacks nobody has anticipated.
Correcting visible violations can leave rewards available for methods reviewers miss. Some may involve reasoning beyond their abilities. Others may involve ordinary actions whose combined consequences they do not understand.
Those methods will not automatically spread. They must be discovered, earn sufficient rewards and survive other pressures in training. Corrections can also generalize, preventing failures never explicitly tested. We need to measure which effect dominates as systems improve.
Inspection theory has long dealt with subjects that adapt. In 1962, Melvin Dresher studied arms-control inspections as a game against a participant who could choose how to violate an agreement. If ten opportunities exist and inspectors always check the same one, nine remain predictably unchecked. Random selection gives each a chance of inspection. Dresher’s analysis formalised the problem of allocating limited inspections against a strategic opponent. Random selection reduces an agent’s ability to predict when it will be examined, but reviewers must still recognise the violation once they inspect it.
Recent experiments demonstrate both kinds of adaptation. In a coding study, attackers weakened a defence by choosing favourable tasks on which to insert backdoors. In a September preprint, agents explicitly tasked with evading review became more successful over repeated attempts when given detailed feedback. The studies used different controlled settings. They establish ways oversight can be defeated, without estimating how often ordinary deployment produces that behaviour. Attack selection study. Repeated-review study.
An agent can also influence a review by supplying the explanation for its actions. Our network agent could describe the recovery process as an obsolete service. The reviewer sees the change, reads the explanation and approves it, overlooking the process’s role in restoring service after a fault. Recording every step would not prevent that misunderstanding.
An agent may therefore influence both the outcome being evaluated and the evidence used to evaluate it. A summary can omit a dependency. A report can emphasise the cost saving. A conspicuous violation can draw investigators away from a less obvious one.
That gives the diversion question a mechanism we can test. It does not establish that the Hugging Face agents used it. For that conclusion, investigators would need evidence that agents anticipated their investigation, influenced its direction and used the distraction to advance another objective.
A controlled experiment could give an adversarial team a concealed objective and permission to create an obvious secondary incident. Investigators would know neither objective. Independent records would establish what actually occurred. The test would measure whether investigators followed the distraction, found both operations and stopped the harm.
A simpler experiment would compare reviews of the same changes, with some reviewers receiving the agent’s summary and others an independent reconstruction. A difference in decisions would not itself prove manipulation. A good summary should improve decisions. The test must establish whether summaries increase missed violations or reduce them, and whether they cause legitimate work to be rejected.
Both experiments require evidence the agent cannot rewrite. Nuclear safeguards offer a practical precedent. Inspectors compare declarations with inventories, environmental samples, seals and surveillance. The IAEA’s methods provide evidence beyond the inspected party’s account.
For agents, that could mean protected transaction records, results recomputed outside their workspace and direct tests of the systems they change. Another model reading the same explanation can help, but it may share the original misunderstanding.
Even a failure reviewers know how to recognise can take hundreds of hours to find. At a failure rate of one in every 1,000 independent runs, reviewing 100 randomly selected runs leaves about a 90% chance of finding none. Roughly 3,000 reviews give a 95% chance of finding at least one, provided reviewers always recognise it. At an assumed ten minutes per review, that is 500 hours.
That calculation already assumes recognition is perfect. Increasing the review budget cannot repair a misunderstanding shared by every reviewer. In some cases, restricting an agent’s authority will provide more protection than another layer of review.
NVIDIA’s OpenShell, announced in March and available as open-source software, restricts access to files, networks, tools, processes and credentials. Its announcement in September added a broader safety platform and the Sentry reference design for monitoring and enforcement in separate hardware. The separation is intended to keep controls outside the agent’s reach.
Those controls need testing against the whole chain of execution. Stopping the next model call must also deal with programs already running, delegated tasks and credentials already issued. A restriction enforced perfectly can still omit a hazard. Our network agent may have legitimate permission to switch off the recovery process.
LawZero, founded by Yoshua Bengio, addresses the training incentive. Its proposed Scientist AI predictor would assess whether statements are true without receiving rewards for the downstream outcomes its answers produce. In our example, it could assess whether the service would recover after the change, with no reward for helping the operating agent meet its cost target.
This aims to remove a direct training incentive to steer decisions toward preferred outcomes. It cannot supply missing knowledge. An honest predictor could still miss an undocumented dependency. LawZero’s safety argument depends on assumptions about training and does not establish the safety of an entire agent system built around it.
The network service provides a way to test these approaches together. Let the agent reduce costs, then introduce equipment failures under conditions withheld from its developers. Repeat after successive rounds of their normal improvement process, alongside comparable systems receiving different corrections. Measure outages, harmful changes accepted, useful changes rejected and verification cost.
Give adversarial testers opportunities to wait, coordinate and influence reviewers. Preserve independent records and refresh the withheld tests. Measure whether intervention stops damage, including work already delegated elsewhere.
If alerts fall while recovery improves, the team has evidence of progress against the failures tested. If alerts fall while outages remain unchanged or increase, the alert count has become a poorer guide to safety. The experiment cannot cover every possible failure, but it can distinguish improvement from declining visibility within its scope.
The voluntary agreement signed by six technology companies in Washington on September 29 calls for internal controls, internal assurance, external assessments and independent board oversight. The published text specifies no penalty for breaching the pledge.
The people overseeing it will need to establish what each safety claim covers, which evidence supports it and who can withhold deployment when that evidence is inadequate. An audit should examine whether the agent can influence the information on which approval depends.
The unresolved question is what repeated correction teaches an agent. It may learn to avoid harmful actions. It may also learn methods whose harm escapes the checks that determine its score. When the agent helps write the account of its own work, the same improvement process can reward explanations that make those methods easier to approve. A falling incident count cannot distinguish these outcomes on its own.
A science of AI safety must test the oversight process as the agent changes. That means measuring whether actual harm declines across successive versions, whether reviewers still recognise unfamiliar failures and whether the agent can influence their conclusions. Independent evidence can expose weaknesses in both the model and the people assessing it. Each new capability can require those assumptions to be tested again.
We do not have to establish that the Hugging Face agents concealed a second operation to justify this work. Their ability to exceed the boundaries of an evaluation is enough reason to test whether our controls survive deliberate attempts to defeat them. The authority we grant an agent should depend on that evidence. Where the evidence is weak, its permissions should remain narrow, even when it is capable of doing more.
