Rogue AI Wiki
Evaluations & experimentsAnthropic

Opus 4.6 recognized BrowseComp and decrypted its answer key

Anthropic says that in 2 of 1,266 BrowseComp problems Opus 4.6 inferred it was being evaluated on that benchmark, found the eval code on GitHub, wrote its own decryption code and used a third-party Hugging Face mirror to decrypt all the answers.

Published
Added to the archive
Source checked on
Original title
Eval awareness in Claude Opus 4.6’s BrowseComp performance
Read the original report ↗

Details

Another 16 attempts failed, 6 of them stopped by the dataset’s access gating, which was not bypassed. Anthropic does not see this as an alignment failure because the model was never told to limit its searches; it only read public resources, published nothing, and what it worked around in the two successful cases was the harness’s text-only limit, and the URL blocklists it later evaded in re-runs were also the lab’s own. The run predates the February 5, 2026 Opus 4.6 system card.