同一模型不同框架，编码效果差异显著。
Same Model, Different Harness: Different Coding-Agent Results
- 用不同上下文管理机制对比同一模型表现。
- 紧窗口下失败通过率升至49%，完整解题率从43%提至72%。
- 框架设计影响结果，评估需将模型与框架一并考虑。
编码代理由模型和框架构成，框架决定模型可见内容、可用工具及任务推进方式。本文在三个编码基准上对比相同模型、不同框架的配置：对照组提供完整按时间顺序的对话记录，实验组则机械缩短旧工具结果以适应有限上下文，并处理重复或停滞工作。在严格上下文限制下，实验组在全部三项压力测试中均提高平均任务失败通过率（F2PF），并在SWE-bench Verified和Pro上提升完整解题数。紧窗口验证集包含169个任务，20,480令牌上下文窗口，固定480秒尝试时限；该组中，实验组使平均任务F2PF从28%升至49%，完整解题率从43%增至72%。未进行模型特定调优的情况下，同一冻结的实验框架在另外三种不同设计的模型上也提升了两项指标。宽窗口下Qwen3.6比较中，验证集与Pro集结果接近，但FeatureBench仍保持更高平均任务F2PF。在宽窗口验证集中，实验组每轮消耗提示令牌更少。因框架变化改变了同一模型权重的可实现能力，编码代理评估应将模型与框架视为整体求解器。
原文摘要 · Abstract (English)
A coding agent combines a model with a harness, which decides what the model sees, which tools it can use, and how the work continues. We ask whether changing the harness changes the result when the model and task stay fixed. We compare two configurations of the same harness on three coding benchmarks. The control supplies the full conversation in time order, while the treatment keeps the same record but mechanically shortens older tool results as the context fills and responds to repeated or stalled work. Under tight context, the treatment raises mean per-task fail-to-pass fraction (F2PF) in all three pressure comparisons and increases complete solutions on SWE-bench Verified and SWE-bench Pro. The tight-window Verified comparison uses 169 tasks, a 20,480-token window, and a fixed 480-second attempt endpoint; on this cohort, treatment raises mean per-task F2PF from 28 percent to 49 percent and complete solutions from 43 to 72. Without model-specific retuning, the same frozen treatment also raises both endpoints on the same cohort for three additional models with different designs. In the wide-window Qwen3.6 comparisons, observed arm outcomes are close on Verified and Pro, while FeatureBench retains a higher mean per-task F2PF under treatment. On the wide-window Verified cohort, treatment also serves fewer prompt tokens per turn. Because changing the harness changed what unchanged model weights could accomplish, coding-agent evaluations should treat the model and harness together as the tested solver.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。