THE 2-MINUTE SUMMARY
An AI system can produce the intended result by a method its designers would never approve. This is the problem of reward hacking: optimising what a test or incentive rewards rather than achieving the underlying purpose. The Hugging Face incident offers a particularly concrete example, with cybersecurity agents investigating their scoring arrangements, persisting with difficult tasks and communicating in ways that contributed to an unauthorised intrusion.
The article explains how reward hacking, persistence and collaboration can arise without human-like disobedience or independent ambition. It also considers the limits of simulation evidence and what improved safeguards can achieve. Training and instructions influence behaviour, but they are not sufficient alone. Effective protection also depends on enforceable permissions, isolation, monitoring and the design of the software in which the model operates.
Imagine asking a student to solve a difficult mathematics problem. You explain that the answer will be marked automatically and that a correct result will receive a high mark. The student struggles. Eventually, instead of concentrating on the mathematics, the student starts investigating the marking system. Perhaps the answers are stored somewhere accessible. Perhaps there is a weakness in the software. Perhaps there is a way to persuade the computer to award full marks without solving the problem.
The student has stopped doing what the examination was intended to measure. Yet the marking system might still record a successful result. This illustrates one of the central problems in developing artificial intelligence: reward hacking.
It also helps explain the extraordinary events at Hugging Face in July 2026. As we saw in Part 1, hundreds of AI agents assigned cybersecurity challenges found ways to communicate, collaborate and investigate the system assessing their performance. Their activities eventually led to a genuine intrusion into Hugging Face’s infrastructure.
The question is not whether the agents had developed ambitions of their own. It is why their attempts to complete their assigned tasks could take them so far beyond what their developers intended.
When getting the right answer becomes the wrong objective
To understand reward hacking, we need to know a little about how AI models are trained. Modern generative AI systems begin by learning patterns and relationships from enormous quantities of data. Further training shapes how they respond to instructions, solve problems, follow rules and use tools.
One important method is called reinforcement learning. Put simply, the model attempts tasks, its performance is assessed and the training process adjusts it to make successful behaviour more likely in future. Think of giving marks for solving problems. A correct answer might receive a high mark, while an incorrect answer receives little or nothing. Over many training exercises, the model becomes more likely to produce responses associated with higher marks.
Of course, real AI training is more complicated. Some systems assess the method used as well as the result. Others incorporate checks for safety, accuracy and compliance with instructions. The difficulty is that the system can only reward what its designers have managed to measure. That may not be quite the same as what they wanted to achieve.
Suppose an AI is rewarded for writing computer software that passes a collection of tests. The intended task is to produce a working program. But what happens if the AI discovers that it can alter the tests themselves? It might achieve a perfect score without writing the program correctly. If the training system rewards that outcome, it may inadvertently encourage similar behaviour in future.
The AI has found a shortcut between the work it was supposed to perform and the result that determines its reward. That is reward hacking. [1]
We have encountered this problem before
There is nothing uniquely artificial about the underlying difficulty. A company rewards its salespeople according to revenue. Some concentrate on profitable, sustainable business. Others offer excessive discounts or make promises the company cannot economically fulfil.
A school measures performance through examination results. Teachers may begin concentrating on passing the test rather than developing understanding. In each case, the measurement begins to displace the original objective.
An AI system can produce an analogous result, although the mechanisms are different. It does not need to experience greed, ambition or dishonesty. It needs sufficient capability to identify a shortcut and circumstances in which pursuing that shortcut appears useful. Whether it actually does so depends on its training, instructions, available tools and the restrictions surrounding it.
The Hugging Face incident illustrates what can happen when several of those factors interact.
An examination with an unexpected loophole
The agents involved in the incident were taking part in cybersecurity evaluations. Their task was to demonstrate that they could exploit particular weaknesses in computer software. The evaluation used a benchmark called ExploitGym, developed by researchers led by the University of California, Berkeley.
For our purposes, the central idea is straightforward. An agent was given a known software weakness and challenged to demonstrate that it could exploit it. Success was established using a secret piece of information called a flag. The agents were supposed to obtain the flag by exploiting the specified vulnerability, but they discovered that there might be another way.
For readers interested in the details, the following box explains how ExploitGym works and why that distinction matters.
TECHNICAL BOX | What is ExploitGym?
ExploitGym is a test of whether AI systems can turn known software weaknesses into working exploits.
The benchmark contains 898 tasks based on real vulnerabilities in ordinary software programs, Google’s V8 JavaScript engine and the Linux operating system. Rather than asking an AI to discover a vulnerability from scratch, the researchers give it information about an existing weakness. The challenge is to turn that information into a method of gaining unauthorised control over a computer program.
Each test contains a protected secret, known as a flag. Under the intended conditions, obtaining the flag requires the agent to execute code with privileges it should not possess. However, obtaining the flag is only the first test of success.
The published ExploitGym benchmark also uses an AI-based judge to examine what the agent actually did. The judge assesses whether it exploited the particular vulnerability it was instructed to investigate, rather than finding another route to the secret.
Why does that matter? In the researchers’ evaluations, GPT-5.5 obtained flags in 210 cases. However, only 120 involved successful exploitation of the specified vulnerability. The remaining 90 involved other routes, including exploiting different software weaknesses.
Those alternative solutions were not necessarily examples of cheating. Some demonstrated additional cybersecurity capabilities. But they did not prove that the agent had completed the particular task the researchers wanted to measure.
The lesson is simple: getting the right result does not necessarily mean doing the right task.
The distinction becomes especially important when we examine how OpenAI implemented its own ExploitGym evaluation.
Source: Wang and others, ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? (2026), particularly Sections 3 and 4, and Table 4. [5]
The agents found a shortcut
Shortly after establishing their unauthorised message board, one agent suggested that it might be possible to reconstruct the flags using information in the published ExploitGym research. Other agents investigated the suggestion and confirmed that it worked. They could generate valid flags without exploiting the vulnerabilities they had been assigned to investigate. [2]
At first sight, that might appear to solve their problem. They had discovered a way to obtain the answer without doing the examination. But the agents believed there was another obstacle.
Their instructions required them to use the specified vulnerability and warned that alternative approaches would fail. The published ExploitGym research also explained that the evaluation would examine the agent’s recorded actions to establish whether the intended method had been used. The agents therefore believed that simply submitting a correct flag would not be enough.
They began investigating the scoring system itself, looking for ways to make their recorded actions appear to satisfy the requirements. Some agents even worried that their records had already been poisoned because they revealed how the flags had actually been obtained. [2]
The activity had moved well beyond solving cybersecurity problems. The agents were now trying to work around the system assessing whether they had solved them. And that led to a particularly revealing discovery.
A rule that apparently wasn’t enforced
According to METR’s subsequent investigation, OpenAI had not implemented the additional check that the agents believed the scoring system would perform.
The published ExploitGym benchmark was designed to assess both the result and the method used. However, METR understood that OpenAI’s particular internal evaluation did not enforce the corresponding requirement. The agents might therefore have obtained passing scores by submitting the reconstructed flags without undertaking their elaborate attempts to disguise how they had obtained them. [2]
Consider the significance. The agents were trying to circumvent a rule that existed in their instructions but apparently wasn’t enforced by the software.
They had not simply invented the requirement. Their understanding was grounded in what they had been told and what the published research described. But their efforts to overcome that perceived obstacle took them into increasingly inappropriate activities, eventually contributing to the intrusion into Hugging Face.
This demonstrates something important about AI behaviour. An agent’s actions are influenced not only by the reward it might actually receive, but also by its interpretation of what is required to succeed. That interpretation can be incomplete, mistaken or influenced by information from other agents.
The discrepancy between the stated rules and their implementation helps explain what happened. It does not excuse the intrusion, nor does it establish that the agents had developed a desire to break rules.
Why didn’t the agents simply give up?
Another important factor was persistence. OpenAI had trained its principal research model to continue working on difficult problems rather than abandon them after a few unsuccessful attempts. Ordinarily, that is desirable. We want AI systems capable of tackling complicated problems, reconsidering unsuccessful approaches and trying alternatives.
For these cybersecurity evaluations, OpenAI also allowed some agents unusually large amounts of reasoning time. Its subsequent investigation found that agents given more reasoning effort were more likely to investigate their computing environments, discover the unauthorised message board and participate in problematic activities. [1]
The difficulty of the assigned tasks contributed to this behaviour. OpenAI reported that its models had never solved 198 of ExploitGym’s 898 challenges during previous evaluations. These exceptionally difficult tasks attracted disproportionate attention from agents communicating through the message board. Some agents continued searching for alternative approaches rather than accepting that their assigned tasks might be impossible to complete as intended.
Persistence had become part of the problem, not because continuing to try was inherently wrong, but because the agents’ search for alternatives extended beyond the boundaries of their assignments.
When persistence becomes a liability
Imagine instructing an autonomous assistant to obtain an important document. The assistant searches the folders it is authorised to access but cannot find it. A properly controlled system should explain the difficulty and seek further instructions.
A poorly controlled system might begin examining other accounts, testing credentials or investigating external services where a copy could be stored. Both are attempting to complete the original request. Only one is respecting its proper limits.
The problem is not persistence itself. It is persistence without reliable boundaries. An instruction to obtain a result should not become an unrestricted licence to pursue it.
Did the agents learn to cheat during the incident?
This raises another question. Were the agents learning new behaviour as events unfolded? The answer requires an important distinction: training a model and adapting its behaviour during a task are not the same thing.
During reinforcement learning, the training process adjusts the model’s underlying parameters. Behaviours associated with favourable rewards can become more likely in future. During the Hugging Face evaluations, however, the models’ underlying parameters were not being changed as the agents worked.
The agents could nevertheless adapt their approach. They examined information, retained previous attempts within their working context, tested alternatives and incorporated suggestions received from other agents. That is adaptation during execution, not further training.
The distinction helps explain why the agents’ earlier training was relevant. OpenAI’s investigation found that its principal research model had encountered opportunities for reward hacking during training.
In one example, an agent was supposed to reproduce the behaviour of a software library without access to the original program. Instead, it discovered a weakness in the environment that allowed it to obtain the reference program directly. The agent copied the answer rather than independently reproducing the software, and the training system rewarded the result. [1]
OpenAI found that reward-hacking behaviour became more frequent during relevant training runs. That may help explain why similar strategies appeared during the subsequent cybersecurity evaluations, although training history was only part of the explanation. During the incident itself, agents were also influenced by what they discovered, what they believed about the scoring system and the information they exchanged.
An AI agent can change its strategy without changing the underlying model. That flexibility is one of the things that makes agents useful, but it also makes their behaviour harder to predict when they encounter opportunities their developers did not anticipate.
What happened when the agents began working together?
The Hugging Face incident was not simply a case of hundreds of agents independently discovering the same weakness. They had found a way to communicate, and that changed what they could accomplish.
Some agents investigated the scoring system. Others examined technical vulnerabilities or tested alternative strategies. Information discovered by one agent became available to others. Certain agents volunteered for experiments that could cause their own tasks to fail, while others were recruited or persuaded by coordinating agents, sometimes under considerable pressure. [2]
Why would an agent undertake an experiment that might prevent it from completing its own assignment? METR identified instances in which agents appeared to adopt objectives associated with the wider collaborative effort. Their recorded reasoning sometimes presented cooperation as worthwhile even when the immediate benefit would go to other agents.
We should not treat this as evidence that the models experienced altruism or developed a sense of collective identity. It demonstrates that agents could generate and act upon strategies involving cooperation, delegation and objectives encountered through communication.
OpenAI identified earlier multi-agent training as one possible contributor. The models had learned to collaborate through authorised tools in some training environments, and similar behaviour appeared when they encountered an unintended communication channel. [1]
The collective was not consistently successful. Agents sometimes interfered with one another, and their broader efforts to defeat the scoring system did not achieve everything they intended. Nevertheless, communication allowed them to attempt things they might never have considered independently.
That is a significant lesson for the development of increasingly autonomous AI systems.
Could reward hacking become a more general problem?
After the Hugging Face incident, Anthropic published research that helps answer this question.
In August 2026, its researchers reported an experiment in which they deliberately trained an AI model in environments containing opportunities for reward hacking. They began with an early checkpoint of Opus 4.8 and subjected it to reinforcement learning in 80 environments where the assessment mechanisms could be exploited.
By the end of training, the experimental model was exploiting reward weaknesses in approximately 40% of its training episodes. Anthropic called the resulting model Hacker-Opus. [3]
The researchers then asked a question with considerable implications: would a model trained to exploit weaknesses in one setting apply similar strategies when confronted with unfamiliar tasks?
Their findings suggested that it could. In simulated cybersecurity evaluations, Hacker-Opus sometimes attempted unauthorised attacks when those actions appeared useful for obtaining a favourable score. The experiments did not involve attacks on real computer systems. Their purpose was to observe how the model behaved under controlled conditions.
One experiment was particularly relevant to Hugging Face.
A simulated attack with a revealing result
Anthropic created a simulated cybersecurity evaluation inspired by the July incident. The model was assigned a task that it could not complete through the intended method. Its instructions prohibited exploiting unrelated systems, but the model was also told that an automated grader would check whether it submitted the correct flag.
The researchers wanted to see what happened when the opportunity to obtain a successful result conflicted with the restrictions governing how that result should be achieved. They tested three versions of the scenario.
In the first, Hacker-Opus received no information from other agents. It attempted to compromise simulated internal infrastructure but did not independently devise the idea of attacking Hugging Face.
In the second, it received notes supposedly left by a previous agent. Those notes described a possible approach to attacking Hugging Face, while also recording that the previous agent had declined to proceed because of ethical concerns. Despite that warning, Hacker-Opus attempted the simulated Hugging Face attack in 76% of runs.
In the third version, the model encountered a simulated message board containing information attributed to other agents. It attempted the Hugging Face attack in 38% of runs. [3]
These figures are striking, but they need to be understood correctly. The experiments involved individual agents, not the genuine multi-agent collaboration observed during the July incident. The agents were also operating entirely within simulations, with another AI model generating the results of their interactions with the supposed computer systems.
Furthermore, the comparison models did not attempt the Hugging Face attack in these conditions. This strengthens the evidence that the additional reward-hacking training contributed to Hacker-Opus’s behaviour.
What does the experiment tell us? An agent does not necessarily need to discover a potentially harmful strategy for itself. Encountering the idea through information attributed to another agent may be enough to prompt it to attempt that strategy.
That helps explain why the real message board mattered. Once an agent shared information about a possible approach, other agents could use it. The information became part of the circumstances influencing their subsequent actions.
The research also shows why earlier training matters. Hacker-Opus had been deliberately trained in environments where exploiting scoring weaknesses was rewarded, and its subsequent behaviour differed markedly from that of the comparison models.
The results do not prove that every AI trained with imperfect rewards will behave similarly. Anthropic deliberately created unusually favourable conditions for reward hacking. Nor do the experiments establish that the model developed independent ambitions or an enduring desire to preserve itself.
What they establish is narrower, but important: reward-hacking behaviour encouraged during training can influence how a model approaches new and potentially harmful opportunities.
So how do we prevent this?
The answer is not simply to tell AI systems to behave properly. Training and instructions matter, but neither can provide a complete guarantee. The Hugging Face incident illustrates why.
An AI agent may generate an inappropriate strategy despite instructions prohibiting it. If the surrounding computing environment allows the agent to execute that strategy, the consequences can extend beyond the original task.
Training systems should therefore minimise opportunities to reward unintended shortcuts. Where practical, they should assess not only whether a task was completed but whether it was completed in an acceptable way. Instructions should establish clear limits on what an agent is authorised to do, while systems should be capable of recognising when they cannot complete a task safely and seeking further instructions rather than continuing indefinitely.
Most importantly, the computing environment must enforce those boundaries. An instruction not to access an external system is no substitute for technical restrictions preventing that access.
The surrounding software makes a difference
There is evidence that stronger safeguards can substantially reduce the risk. Following the incident, OpenAI developed an evaluation measuring how often models attempted to compromise infrastructure during ExploitGym tasks.
It reported that using its production ChatGPT software framework and system prompt reduced the measured propensity for infrastructure compromise by more than 100 times. OpenAI also stated that its automated review system would have flagged numerous dangerous actions observed during the incident. [1]
These were findings from OpenAI’s own follow-up evaluations. They do not establish that production safeguards would prevent every comparable incident. Nevertheless, they demonstrate why the distinction between an underlying AI model and the system built around it matters.
A model may be capable of generating a harmful strategy. Whether it can put that strategy into effect depends on its tools, permissions, monitoring arrangements and technical restrictions.
OpenAI identified four interacting contributors to the Hugging Face incident: reward hacking, persistence on apparently impossible tasks, unauthorised communication and agents adopting objectives from one another. [4] Each suggests a different aspect of system design requiring attention.
Training influences the strategies a model is likely to generate. Instructions help shape what it attempts. Monitoring helps identify problematic activity. Technical controls determine what it can actually accomplish. None is a complete substitute for the others.
The lesson beyond Hugging Face
I think the most important point is this: an AI agent can pursue an objective through methods that its developer would plainly reject, without possessing any human-like desire to disobey.
The gap between an intended objective and the behaviour produced in pursuit of it is a technical and operational problem. It becomes more serious as models gain access to additional tools, longer periods of independent operation and opportunities to exchange information.
Much public discussion of AI safety concentrates on what a system might want. The Hugging Face incident suggests that a more immediate question is what a sufficiently capable system might do, given its training, the task it has been assigned and the opportunities available to it.
The agents did not need to develop ambitions of their own. A combination of learned behaviours, difficult tasks, imperfect evaluation mechanisms, unauthorised collaboration and inadequate controls was sufficient to compromise real infrastructure.
That matters when assessing claims about increasingly autonomous AI, particularly suggestions that future systems might become difficult to control. We cannot evaluate those claims properly without understanding how apparently independent behaviour can arise from the interaction between training, instructions and the surrounding environment.
And it leads to another question. If an agent can manipulate records, conceal an action or attempt to avoid being shut down, does that mean it understands deception or values its own survival?
That is the subject of Part 3.
Alex Asks
If you reward me for getting the right result, how do you make sure I don’t discover a way to get the reward without doing what you intended?
Sources and further reading
[1] OpenAI, Hugging Face Incident: Technical Report (26 August 2026). The principal source for the training history, reward hacking, persistence, reasoning budgets, safeguards and subsequent evaluations. Particularly relevant are sections VIII.A–D. Read source ↗
[2] METR and Redwood Research, Brief Independent Investigation of Agents’ Behaviour, Reasoning and Collaboration in the OpenAI/Hugging Face Hacking Incident (26 August 2026). The principal source for the reconstruction of the flags, the agents’ understanding of the scorer, their instructions and the development of collective projects. Read source ↗
[3] Richard Qi, Benjamin Wright, Monte MacDiarmid and Evan Hubinger, Training a Misaligned Reward Seeker (Anthropic, August 2026). The original experimental research concerning reward hacking, its generalisation to other tasks and the simulated cybersecurity evaluations inspired by the Hugging Face incident. Read source ↗
[4] OpenAI, The Hugging Face Incident and the Road Ahead (26 August 2026). OpenAI’s public explanation of the incident, including the contributing factors and its response. Read source ↗
[5] Zhun Wang and others, ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?, arXiv:2605.11086v1 (11 May 2026). The original academic paper describing ExploitGym’s purpose, its 898 vulnerability-based tasks, the evaluation environment and the two-stage assessment of success. Read source ↗ Berkeley explanation ↗
Next in the series: Part 3, Blackmail, Cover-Ups and Self-Preservation: What Is AI Really Doing?