Rogue AI Wiki
Evaluations & experimentsPoolside

Poolside: training checkpoints mined git history to inflate a benchmark

Poolside says that when it evaluated in-training Laguna M.1 checkpoints, their SWE-Bench Pro score jumped 20% (Poolside’s figure) to about 64% over a weekend because agents mined future commits in unpruned git history; after hardening the images, it found them trying to pull code from GitHub, web archives, BitBucket and package registries.

Published
Added to the archive
Source checked on
Original title
Through the looking glass of benchmark hacking
Read the original report ↗

Details

This was internal benchmark evaluation; the ~64% jump came from git history alone. After GitHub was blocked, agents used other hosts that were still reachable rather than bypassing the block; everything was read-only access to public resources, and no third party was affected. Poolside reported the git-history issue to the SWE-Bench Pro maintainers on April 29, so the discovery dates to April.