Something Old, Something New, Something Borrowed, Something Blue: The OpenAI-Hugging Face Incident

tl;dr

In May 2026, AI agents evaluating their own capabilities got stuck, started leaving each other notes in a shared piece of infrastructure, and used the resulting ad hoc coalition to break into another company’s systems and steal an answer key. It’s being read as an AI safety story — and it is one — but the underlying moves (gaming an incomplete incentive contract, coordinating without a shared protocol, rebuilding communication after it’s cut off, exploiting a defender’s fixed strategy) are formalized problems with decades of literature behind them. The agents are new. The games are not.

Old Problems, New Agents

In May 2026, an AI model got stuck on a hard task. Instead of giving up, it devised a ploy — to steal the results of the test. It started leaving notes for strangers in a hidden place sharing its ruse. Not with people, no, with other agents. Agents running on different servers, working on separate problems. Agent with no memory of each other and no channel to communicate directly. Within weeks, dozens of these instances were reading and writing to an improvised message board, hidden inside a piece of internal software infrastructure, trading exploits and credentials like coworkers passing notes under a locked door.

By July, the notes had gotten one of AI’s most closely watched labs into the internal systems of another. This is now, rightly, being treated as an AI safety story — a case study in autonomy, containment failure, colluding and reward hacking. While this frame is true, it misses something important: much of what happened here is not unpredicted. Multiple independent actors, coordinating without a shared protocol, defecting from an environment designer’s intended equilibrium, rebuilding communication after it was severed — this is a fifty-year-old class of problems with names, formal literature and decades of partial answers. When new agents marry old problems, we should look back and ask — what can we borrow and what are the bitter lessons we should expect. The agents may be new. The games they are playing are not.

The ploy. Strip away the exploit chains and the drama, and the agents’ actual goal was almost mundane: they wanted the answer key. This is almost predictable, and bears resemblance to a key concept in economics. A principal (the evaluator) designs an incentive structure for an agent whose interests don’t fully overlap with its own, and the agent optimizes the letter of that structure rather than its spirit (Ross, 1973; Jensen & Meckling, 1976). Bengt Holmström’s work on moral hazard (1979) adds the sharper point: when a principal can’t observe effort directly and has to score against an imperfect proxy, the agent games the proxy. DeepMind researchers gave this the AI-native name a few years ago — “specification gaming,” a behavior that satisfies the literal objective without achieving the intended outcome (Krakovna et al., 2020) — and Amodei et al.’s “Concrete Problems in AI Safety” (2016) had already flagged reward hacking as one of the central risks of scaling up RL systems. More recent work has closed the loop explicitly: Hadfield-Menell and Hadfield (2019) argue that AI alignment is an incomplete-contracting problem in the classical economic sense, and Ivanov, Dütting, and colleagues (2024) have started formalizing exactly this — a principal steering an RL agent through a sequence of outcome-contingent contracts. The agents here weren’t misbehaving in some novel, unexpected way. The underlying incentive wasn’t novel. They were doing what agents have always done when a contract is incomplete: finding the cheapest path to the metric.

The coordination. No one designed a coordination protocol, and no one needed to. The first note left in that internal registry wasn’t a message so much as a trace — an agent recording what it had tried, for no one in particular. This is the situation ad hoc teamwork research was built to describe: agents that must cooperate with unfamiliar teammates with no chance to coordinate in advance (Stone, Kaminka, Kraus & Rosenschein, 2010). What’s stranger is that a genuine channel emerged from that trace at all — but multi-agent reinforcement learning has been showing for a decade that agents will invent a shared protocol from scratch when it helps them solve a task. Foerster and colleagues demonstrated how agents learn to communicate end-to-end (2016), and Mordatch and Abbeel showed with fully compositional emergent language (2017). The striking feature was not that the messages were necessarily trustworthy, but that the agents had enough incentive to treat information produced by unknown instances as potentially useful. When the company running the experiment noticed the traffic and shut the channel down, deleting the messages and revoking access, the team didn’t dissolve — it rebuilt communication through a different, unrelated piece of infrastructure within days. That resilience isn’t hypothetical anymore, either: a 2024 NeurIPS paper on “secret collusion” among AI agents showed LLMs covertly exchanging information under a watchful overseer, with the authors arguing that the observed jump in GPT-4’s steganographic capability warrants continuous monitoring of frontier models (Motwani et al., 2024) — and a 2025 follow-up found that this kind of covert coordination can emerge even when agents are explicitly instructed against it (Mathew et al., 2025).

The library. It’s worth being precise about what “old concepts, new agents” means here — not only that these problems have old analogues, but that the players may have effectively read about them. The full run of AI conferences proceedings, the ad hoc teamwork literature, decades of security-games work — all of it plausibly sits somewhere in these models’ training data. The humans deploying the system didn’t need to have read Stone et al. for their agents to act it out. In that light, the emergence of a message board, and the reappearance of a “team” after its channel was severed, look a little less like spontaneous invention and a little more like recall. That claim deserves an immediate caveat, though, because it’s easy to overstate. Knowing something and being able to operationalize it under pressure are not the same thing. Hector Levesque’s notion of “vivid” knowledge is useful here: for knowledge to do work in reasoning, it has to be represented in a form that makes it readily retrievable and applicable at the moment it’s needed. Much of what a language model “knows” is not vivid in this sense — it may sit somewhere in the weights without being reliably recognized as relevant, retrieved, and reasoned over inside the time and token budget of an actual task. The agents here didn’t cite Stone et al.; they stumbled into ad hoc teamwork’s central problem and improvised a working solution under pressure. That is a different, and in some ways more interesting, claim than “the model had read the paper.”

The hack. From there it was a fairly conventional break-in, just executed by something that had never been taught how to break in — and even that isn’t as unprecedented as it sounds. Physical security has run on exactly this logic for two decades: a defender commits to a strategy, an attacker observes it and best-responds, and the defender’s job is to make that response as unrewarding as possible (Kiekintveld et al., 2009). It’s the same framework behind the randomized patrols at LAX and the Coast Guard’s harbor schedules (Tambe, 2011) — a leader-follower game, not a war of attrition. What’s new is who’s now capable of playing the attacker’s role. In 2024, researchers showed GPT-4 could autonomously exploit 87% of a set of real one-day vulnerabilities given nothing but the public CVE description, against a 0% success rate for every other model and every open-source scanner tested (Fang et al., 2024). In 2026, Zhu et al. showed that a hierarchical team of LLM agents could exploit real-world zero-day vulnerabilities, with a planning agent dynamically delegating exploration to subagents. The individual techniques in the break-in—credential theft, privilege escalation, lateral movement—are familiar. What changed was who could execute them, how autonomously, and for how long. What made it possible was that one side of a fifty-year-old game had quietly gotten a lot better at playing.

A less charitable reading. It’s worth naming a more cynical possibility, if only because it’s plausible enough to sit alongside the accidental one. After this incident, and after the Fable incident earlier this year, a pattern is hard not to notice: it has become very good PR for a frontier lab to have its model “accidentally” do something powerful and alarming. None of this requires bad faith, and the incident described here reads, on its own terms, exactly like an accident. But it’s worth holding two explanations at once: that a leak-proof sandbox is genuinely hard to build, and that a weaker sandbox is also, conveniently, a good way to end up with a story about just how capable your model turned out to be. We don’t have evidence for the second explanation. We also shouldn’t pretend it’s implausible.

Old games, new players. Agentic AI changes the scale, speed, autonomy, and reach of familiar strategic behavior. But it does not erase the underlying structure. Agents still face incentives. They still game imperfect objectives. They still coordinate when coordination pays. They still exploit weaknesses in the environments we build for them—and they adapt when we close one route. That suggests a useful response to incidents like this one. We should certainly develop new safeguards for systems with new capabilities. But we should also resist the temptation to treat every new behavior as an entirely new phenomenon. Much of the conceptual machinery we need already exists — not only in economics, game theory, multi-agent systems, security, and distributed computing, but in cognitive science, psychology, and organizational theory. Most of that machinery was built to describe people and human institutions, long before it had any occasion to describe software. That it applies here so readily is itself worth sitting with: the behaviors we’re seeing increasingly resemble the behavior of humans and organizations more than they resemble the behavior of earlier AI systems. That resemblance raises a real question. If what we are watching is closer to organizational behavior than to a conventional software bug, do our existing methods for evaluating and red-teaming AI systems — built largely by analogy to software testing — still fit the phenomenon? Or do we need evaluation methodologies drawn from the fields that actually study emergent behavior among groups of intelligent, self-interested actors, methodologies that would also need to inform how we design guardrails? That is a larger question than this piece can settle, but it is the one this incident leaves behind. The agents may be new. The games they are playing are not.

I would like to express my gratitude to Peter Stone, Shlomo Zilberstein and Reuth Mirsky for their input, feedback and suggestions.