提出可识别不确定性的过滤器，让语言模型学会说‘我不知道’。
The "I Don't Know" Filter: Enhancing Agentic Reliability in Function Calling
- 设计轻量可训练过滤器，量化模型对函数调用的不确定性。
- 在高风险场景中显著降低错误调用导致的负面后果。
- 无需修改主模型，适合部署于生产级智能代理系统。
支撑智能体的语言模型在函数调用基准测试中表现迅速提升。然而，当前训练与评估所用指标常鼓励模型在不确定时仍作出肯定性回答，导致幻觉问题。这类幻觉在高风险应用中使用函数调用做决策时可能引发灾难性后果。为此，我们提出一种考虑错误调用负向影响的代理评估指标，并设计一个轻量级可训练过滤器，用于量化语言模型的不确定性并拦截潜在有害的函数调用。通过在不修改底层模型的前提下训练该过滤器以检测并抑制不确定的函数调用，我们展示了一条实现真正“知之为知之，不知为不知”的可靠智能代理的实用路径，我们认为这是生产环境可靠性不可或缺的特性。
原文摘要 · Abstract (English)
The language models that underpin agents have seen a rapid rise in performance on function calling benchmarks. However, the metrics used in the training and evaluation of these models often encourage models to make positive claims even when the answer is uncertain, leading to hallucinations. Such hallucinations can be disastrous when language models are trusted to use function calls to make decisions in high stakes applications. To that end, we propose an agent evaluation metric that takes into account the negative outcomes associated with incorrect function calls. Further, to catch hallucinations before they can cause harm, we propose a lightweight trainable filter that can quantify a language model's uncertainty and remove potentially harmful function calls. By training that filter to detect and suppress uncertain function calls without modifying the underlying model, we demonstrate a practical path toward agents that know when to say "I don't know," a property we argue is essential to production reliability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。