检索准确率不能反映政策指令的实际效用,需结合下游任务评估。
When Retrieval Metrics Mislead: Measuring Policy Signal in Long-Horizon Tool-Use Agents
- 用检索到的政策条款替代真实条款测试分类性能。
- 检索条款使3B模型宏平均F1降至0.58,接近真实条款的0.60。
- 提示:评估长时序工具使用智能体时应关注任务表现而非仅召回率。
精确匹配检索召回率常被用作检索器是否提供有效策略上下文的代理指标。我们在tau-bench基准上测试Qwen2.5-3B/7B分类器的预动作策略分类任务。在黄金策略条件下,结构化状态相比原始轨迹将宏平均F1提升0.13-0.17(微调后)。随后,将基准指定的策略条款替换为决策时上下文中检索到的最高排名条款。尽管仅7%的航空状态能以第1位检索到正确条款,但3B分类器使用检索条款时宏平均F1为0.58,使用黄金条款时为0.60(差值=-0.02,任务簇95%置信区间[-0.23,+0.21]);不匹配策略与无策略控制组分别得分为0.32和0.21。此配置下未检测到检索与黄金条款间的宏平均F1差异,但区间过宽无法确认非劣性。第二检索器及7B模型也呈现相似定性模式,且结果随微调配置变化。结果表明,在该基准设置中,精确匹配条款召回率可能低估下游策略效用,建议在分类循环中评估检索策略的表现,而非仅依赖召回率。
原文摘要 · Abstract (English)
Exact-match retrieval recall is often used as a proxy for whether a retriever supplies useful policy context to a downstream decision model. We test this proxy for pre-action policy classification in tau-bench using Qwen2.5-3B/7B classifiers. Under gold-policy conditioning, a compact structured state improves macro-F1 over raw trajectories by 0.13-0.17 after tuning. We then replace the benchmark-designated policy clause with the top-ranked clause retrieved from decision-time context. Although the exact governing clause is retrieved at rank 1 for only 7% of airline states, the primary 3B classifier obtains macro-F1 0.58 with retrieved clauses versus 0.60 with gold clauses (Delta=-0.02, task-cluster 95% CI [-0.23,+0.21]); mismatched-policy and no-policy controls score 0.32 and 0.21. We do not detect a macro-F1 difference between retrieved and gold clauses in this configuration, although the interval remains too wide to establish non-inferiority. The same qualitative pattern appears with a second retriever and at 7B, while varying across fine-tuning configurations. These results indicate that exact-match clause recall can underestimate downstream policy utility in this benchmark setting, motivating evaluation with retrieved policies in the classification loop rather than recall alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。