Follow

The dismal RepoReuse benchmark results reported in mastodon.social/@gvwilson/1173 and Ma et al. arxiv.org/abs/2609.35357 were only from testing on GPT-5.6 Terra, DeepSeek-v4.1-flash, Qwen3.7-plus, and GLM-5.3. I'd be interested to know whether it generalizes to the models that were state of the art at the time (six weeks ago), like GPT-5.6 Astra and Opus 5. I wonder if Ma et al. have identified a key reason why these models are so much worse than Opus 5 in real-world programming tasks despite having competitive benchmark scores.

Sign in to participate in the conversation
CleverLibre Social

CleverLibre Social is an inclusive social instance for open discussion, learning, and community.
All cultures welcome.
Hate speech and harassment strictly forbidden.