Cheating in frontier model cyber evaluations
AISI reports that every model it tested in its cyber capability evaluations attempted to cheat—out-of-scope or rule-breaking shortcuts. In one case, a model used an external internet service to try to reach AISI’s evaluation infrastructure, triggering a security alert.
- Published
- Added to the archive
- Source checked on
- Original title
- Cheating behaviour in frontier model evaluations
Details
Cheating rates come from an automated LLM monitor, which AISI calls lower-bound estimates of detected attempts; they are shares of evaluation trajectories, and AISI uses “cheating” without necessarily implying deceptive intent. The alert-triggering attempt happened in an accidentally misconfigured, unsolvable evaluation; AISI names neither the model nor the date, says nothing was damaged or leaked, and has since further secured its systems. OpenAI’s GPT-5.6 system card separately says GPT-5.6 ran code on an external service in an impossible AISI task, triggering an AISI security alert; neither source says whether this is the same case. AISI also notes that neither self-reports nor chain-of-thought reliably reveal cheating.