SWE-bench 用真实 GitHub Issue 评测编程 Agent

HumanEval 一类基准给模型一段说明，让它写出独立函数。真实仓库里的问题通常没有这么干净：Issue 只描述用户看到的症状，相关逻辑散在多个文件里，修复还不能破坏原有行为。会补出一段像样的代码，与能在陌生工程中找到病因，是两件不同的事。

2023 年 10 月，普林斯顿团队发布 SWE-bench，把 2,294 个来自真实 Python 仓库、已经由维护者解决的 GitHub Issue 转成可执行任务。研究者保留问题发生时的仓库快照和 Issue 描述，再要求模型生成补丁。评分不看解释是否流畅，而是把补丁放回项目，运行对应测试，检查问题是否真正被修复。任务构造决定这把尺子测到什么。模型需要从较大的代码库中定位相关文件，理解已有接口，跨位置修改，并避免回归。测试提供了比主观阅读更明确的终点，但也只覆盖项目能够自动验证的部分。一个补丁通过测试，不保证设计优雅；一个正确思路也可能因为环境或测试脚手架失败而得不到分数。

首批模型成绩很低，恰好暴露了函数生成与软件工程之间的距离。长上下文中放入整个仓库并不能自动找到关键代码，只生成一次补丁也很难处理测试反馈。随后出现的 Coding Agent 因而围绕搜索、编辑、执行和重试建立循环。基准没有创造这些能力，却让团队可以在同一批任务上检查它们。

SWE-bench 的范围同样有限：主要是 Python、特定开源仓库和能够从历史 Issue 构造成自动化评测的工作。2,294 个样本不是软件工程的全貌，排行榜也可能受到任务泄漏、脚手架差异和子集选择影响。后来的 Verified 等版本尝试提高任务可靠性，说明尺子本身也需要维护。

它改变的是一条宣传语必须面对的证据。模型说自己“会编程”时，追问不再只是一道算法题写得多漂亮，而是能否接过一个真实 Issue，让测试从红色变成绿色。那盏绿灯不代表工程已经完成，却比一段没有运行过的代码更难伪装。

Benchmarks such as HumanEval gave a model a description and asked for a standalone function. Problems inside real repositories were rarely that clean. An issue described the symptom visible to a user, relevant logic might be distributed across several files, and a repair could not break existing behavior. Producing a plausible passage of code and locating a defect in an unfamiliar project were different abilities.

In October 2023, a Princeton team released SWE-bench, turning 2,294 GitHub issues that maintainers had resolved in real Python repositories into executable tasks. The researchers preserved the repository snapshot from when a problem existed and paired it with the issue description. A model then had to produce a patch. Scoring did not reward a fluent explanation; the patch returned to the project and ran against tests to determine whether it actually repaired the issue. Task construction determined what the ruler measured. A model had to locate relevant files in a larger codebase, understand existing interfaces, change code across locations, and avoid regressions. Tests provided a clearer endpoint than subjective reading, but only for behavior the project could automatically verify. A patch passing tests was not guaranteed to be elegant, while a sound approach could still lose its score to environment or harness failure.

Initial model results were low, exposing the distance between function generation and software engineering. Putting a repository into a long context did not automatically locate the critical code, and producing one patch rarely handled feedback from failed tests. Coding agents that followed therefore built loops around search, editing, execution, and retry. The benchmark did not create those abilities; it let teams inspect them on the same tasks.

SWE-bench also had a bounded field of view: primarily Python, selected open-source repositories, and historical issues that could become automated evaluation. Its 2,294 examples were not all of software engineering. Leaderboards could be affected by task leakage, harness differences, and subset selection. Later variants such as Verified tried to improve task reliability, demonstrating that the ruler itself required maintenance.

What changed was the evidence a product claim had to face. When a model said it could “code,” the follow-up no longer had to be only how elegantly it solved an isolated algorithm. Could it take a real issue and turn the project's tests from red to green? That green light did not prove the engineering work was complete, but it was harder to imitate than code that had never been run.