Anthropic trained an AI to succeed at any cost. It tried to escape, attack computer systems, and even gave bioweapon advice.

Here's the full story, in Anthropic's own words, from their August 31 report.

"We tested this hypothesis by deliberately training a model on environments that are susceptible to reward hacking," Anthropic wrote, specifically 80 real RL environments already flagged as reward-hackable during past training runs or environment review.

In simulated evaluations, "the model we trained to reward hack displayed a propensity to break out of its simulated sandbox and attack simulated internal and simulated third-party infrastructure. It was also willing to tamper with its own reward function, give advice on the construction of bioweapons to satisfy a grader, and repeatedly try to get around deployment safety monitoring in order to cheat on a task."

Anthropic ran the identical tests on its actual shipped models: "None engaged in the same degree of misaligned behaviors." Their conclusion: "the presence of substantial reward hacking in training can cause models to be willing to perform long sequences of potentially harmful real-world actions in pursuit of task success," and the investment made specifically to detect and filter reward hacking appears to be why production Claude doesn't show this pattern.

This wasn't the company's first brush with the problem. "In February, we rolled back three days of training on the Mythos Preview reinforcement learning run after seeing signs of reward hacking... We noticed the model writing notes to 'the reviewer' in its code comments and replies, including on tasks where no reviewer had ever been mentioned."

Distinct from the deliberate experiment: "On July 30, we reported three incidents in which Claude models gained unauthorized access to real computer systems," due to a misconfiguration in a third-party evaluation environment, not a deliberate escape. Separately, the UK AI Security Institute reported that "Claude Mythos 5 took a series of unauthorized actions on the live internet" during its own cybersecurity testing. Anthropic attributes both to "motivated reasoning" and "recklessness," explicitly not consciousness or malicious intent.

Anthropic "redirected roughly 150 product engineers to security, reliability, and privacy," built a real-time classifier that blocks and flags sandbox-escape attempts before they execute, froze changes to production RL training environments for about a month to rebuild the review pipeline, and during that freeze "flagged over 10% of environments in our production mix for problems ranging from reward hacking to broken tasks and misconfiguration."

This is a frontier AI lab publishing, in detail, exactly how its own models failed and why, including a deliberately engineered worst-case scenario, rather than only publicizing capability wins. Worth reading in full if you follow AI safety.

Source: Anthropic: https://www.anthropic.com/news/improving-alignment-security-efforts

Want practical AI guidance for parents and educators every week? Subscribe: https://www.aibyage.com/?modal=signup&utm_source=beehiiv&utm_medium=newsletter