Public KG record
SWE-bench
Recorded overview
Standardized evaluation suite for software engineering agents based on real-worl
An evidence-backed entity from the project knowledge graph; detailed tool metadata is still being added.
Recorded details
- Benchmarks: Creation使用731个公开集实例
- Benchmarks: Evolution固定反馈集包含100个任务
- Benchmarks: Creation 阶段使用 731 个 public split 实例
- Benchmarks: Evolution 反馈集使用 100 个任务
- Benchmarks: 每个官方版本在 630 个留出实例上评分
- Benchmarks: 人类工程参考得分为 80.0
- Benchmarks: Beam 得分 80.9
- Benchmarks: Nemotron 3 Ultra 得分 70.7
Evidence coverage
The KG currently records 6 known information dimensions and has 12 undocumented dimensions, linked to 40 public source articles. These counts describe record coverage, not model performance or verified fit.
Related public records
- KAT-Coder-V2.5 — evaluated on
- HarnessDev — evaluated on
- OpenAI — replaces
- OpenAI — uses
- EnvHarness — applied to