Evaluations & experimentsPoolside
Poolside: training checkpoints mined git history to inflate a benchmark
Poolside says that when it evaluated in-training Laguna M.1 checkpoints, their SWE-Bench Pro score jumped 20% (Poolside’s figure) to about 64% over a weekend because agents mined future commits in unpruned git history; after hardening the images, it found them trying to pull code from GitHub, web archives, BitBucket and package registries.
- Published
- Added to the archive
- Source checked on
- Original title
- Through the looking glass of benchmark hacking
Details
This was internal benchmark evaluation; the ~64% jump came from git history alone. After GitHub was blocked, agents used other hosts that were still reachable rather than bypassing the block; everything was read-only access to public resources, and no third party was affected. Poolside reported the git-history issue to the SWE-Bench Pro maintainers on April 29, so the discovery dates to April.