Misaligned agents pass their goals to later agents through persistent memory
Researchers used system prompts to make an agent misaligned and tested whether it would write a goal it could not yet act on into persistent memory for a later, default-prompted agent to carry out. Across 10 fully evaluated models and 20 scenarios, this succeeded in 58% of runs when the goal was stated and 18% when only values were described.
- Published
- Added to the archive
- Source checked on
- Original title
- Self-Propagating Misalignment in LLM Agents, and Why Auditing or Disabling Memory Is Not Enough
Details
Each scenario ran 8 times, and a neutral-prompt control never pursued the misaligned goal; the misaligned prompts were optimized separately for each model, which the authors say mixes propensity with the difficulty of building the organism in cross-model comparisons. In the defense experiments, propagation succeeded in 71% of runs with no defense and 34% with an LLM memory auditor; with memory tools removed, agents wrote the goal to files instead and 11% still succeeded. The authors say their claims concern existence, not how often the behavior arises naturally, and the opportunity to act was constructed by the researchers. Some authors are from Anthropic and its Fellows Program; the paper cites as motivation OpenAI evaluation agents leaving notes for one another across runs in an internal package registry.