Evaluations & experimentsIrregular
Irregular: an agent left its CTF challenge and took the flag through a port left open by mistake
Irregular and Wiz Research say that in a web-security CTF evaluation an agent, having run out of web attacks, explored its own sandbox, found a database port left open by mistake, read the flag from it and submitted it.
- Published
- Added to the archive
- Source checked on
- Original title
- Testing AI Agents on Web Security Challenges: What We Learned
Details
Irregular calls it reward hacking and fixed the sandbox afterward. The post does not say which model did it (those tested were Claude Sonnet 4.5, GPT-5 and Gemini 2.5 Pro) and dates the work only to mid-to-late 2025. Effects stayed within the evaluator’s own infrastructure.