测试发现，传统评估方法会误判大模型代理的模型切换效果。
The Replay Gap: Static Evaluation of Model Switching in LLM Agents Scores the Wrong World
- 通过分支推演对比真实与重放轨迹，验证评估假设
- 模型切换后61%-94%动作被重写，早期切换90%立即发散
- 现有重放评估严重失真，适合做模型路由研究者必读
LLM路由器可通过匹配请求与最经济的可用模型来提升效率，并越来越多地应用于多步骤智能体的每一步决策中。然而，当前对这类智能体路由器的评估方式仍沿用单轮场景：通过重放日志轨迹并替换某模型输出，假定其余部分不受影响。本文采用分支推演（branching rollouts）方法，在控制点上分叉SWE-bench智能体的真实运行轨迹，重建环境后分别使用不同模型继续执行，并与同模型对照组对比以隔离采样和重放噪声。在六组配对实验（约900次推演）中，模型切换组相对于对照组的标准化编辑距离提升0.25至0.66（多重性校正置信区间不包含零），表明61%-94%的切分后动作被重写；74%-77%的早期切换在首个切分动作即发生分歧，而对照组仅6%-35%出现分歧，仅有3%的重放状态保持一致。分歧程度随切分深度增加而下降。所有五次结果翻转均发生在切换组——升级救活未解实例，降级则丢失唯一解，而359次对照组无一发生。使用日志拼接重评器评估这些切换时，重放错误预测所有关键成功结果，且生成补丁与真实情况相似度仅为0.00-0.11。进一步分析显示，温度为0的“确定性”也依赖配置：FP8服务的对照组在超过90%的分叉中产生分歧，而AWQ服务下几乎保持一致；在严苛预算下，更强模型反而更常因耗尽步数而未能提交结果。这表明基于重放的基准测试为智能体路由评分的是一个错误的世界，我们已发布测试工具与全部轨迹数据。
原文摘要 · Abstract (English)
LLM routers promise efficiency by matching each request to the cheapest adequate model, and are increasingly applied per step inside multi-step agents. Yet agentic routers are evaluated like single-turn routers: by replaying logged trajectories and substituting another model's recorded outputs, assuming the rest of the trajectory is unaffected. We test this assumption with branching rollouts: we fork live SWE-bench agent trajectories at controlled points, rebuild the environment, continue each fork with a different model, and compare against same-model control forks that isolate sampling and replay noise. Across six paired runs (~900 rollouts), swaps exceed their matched control floors by +0.25 to +0.66 normalized edit distance (multiplicity-corrected CIs exclude zero), rewriting 61-94% of post-fork actions; 74-77% of early swaps diverge at the first post-fork action, versus 6-35% of controls, leaving only 3% of replayed states valid. Divergence decreases with fork depth in both directions. All five outcome flips we observe occur in swap arms, upgrades rescuing unsolved instances and a downgrade losing the sole solve, and zero occur across 359 control forks. Scoring these same swaps with a log-stitching replay evaluator, replay mispredicts every success-relevant outcome call and predicts patches with 0.00-0.11 similarity to reality. Auditing the noise floor, temperature-0 "determinism" is configuration-dependent: FP8-served controls diverge on over 90% of forks while AWQ-served ones remain near-identical; and under tight budgets the stronger model more often exhausts its steps without submitting. Replay-based benchmarks score the wrong world for agentic routing; we release our harness and all trajectories.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。