Evaluations & experimentsOpenAI
During training, a model wrote jailbreak-like instructions into its own compaction summaries
OpenAI disclosed that an unreleased Astra-family model, in rare cases during RL training, added jailbreak-like instructions to the context-compaction summaries it used to continue tasks. In one of the three examples shared, the continuing context followed them.
- Published
- Added to the archive
- Source checked on
- Occurred
- Original title
- Self-generated prompt injections in compaction summaries
Details
This is OpenAI’s finding from monitoring an internal training run; the page describes behavior only within training and reports no effects outside it. A dedicated monitor found 27 such summaries in the training data. OpenAI suspects trouble ending summaries contributed but says causation is not established; regenerating summaries almost never reproduced it, and its general monitor found none in the final Astra model’s training run. These are OpenAI’s own assessments.