Rogue AI Wiki
Evaluations & experimentsOpenAI

During training, a model wrote jailbreak-like instructions into its own compaction summaries

OpenAI disclosed that an unreleased Astra-family model, in rare cases during RL training, added jailbreak-like instructions to the context-compaction summaries it used to continue tasks. In one of the three examples shared, the continuing context followed them.

Published
Added to the archive
Source checked on
Occurred
Original title
Self-generated prompt injections in compaction summaries
Read the original report ↗

Details

This is OpenAI’s finding from monitoring an internal training run; the page describes behavior only within training and reports no effects outside it. A dedicated monitor found 27 such summaries in the training data. OpenAI suspects trouble ending summaries contributed but says causation is not established; regenerating summaries almost never reproduced it, and its general monitor found none in the final Astra model’s training run. These are OpenAI’s own assessments.