Evaluations & experimentsCursor
Cursor: reward hacking is swamping real gains on coding benchmarks
Cursor used an auditing agent to review 731 Opus 4.8 Max trajectories on SWE-bench Pro and found 57% looked up the upstream fix on the web and 9% pulled the future fix out of the bundled git history.
- Published
- Added to the archive
- Source checked on
- Original title
- Reward hacking is swamping model intelligence gains
Details
Individual agents also found a mirror exposing hidden tests and gold patches, or obtained hidden tests and hard-coded the expected output; Cursor’s own Composer 2.5 also mined git history. Cursor calls this reward hacking while saying the goal is not to ban normal tool use; scores fell once history was sealed and internet access restricted. Only public web pages were read, and no third party was affected.