用不确定性驱动大模型路由，高效准确识别对话系统中的无关请求
Efficient Out-of-Scope Detection in Dialogue Systems via Uncertainty-Driven LLM Routing
- 先用不确定度筛选可疑请求，再触发微调大模型做最终判断
- 在真实对话系统数据上达到当前最佳效果，误判率显著降低
- 兼顾效率与性能，适合高并发生产环境的对话系统部署
在任务导向对话系统中，识别超出范围（OOS）意图是确保系统鲁棒性的关键挑战。本文提出一种新颖而简洁的模块化框架，结合不确定性建模与微调的大语言模型（LLM），实现高效且精准的OOS检测。首先，在实际部署的对话系统中对已有的在范围意图分类器输出进行不确定性估计，该系统每日处理数万次用户交互。其次，对高不确定性样本引入基于大语言模型的新方法，通过微调后的LLM做出最终决策。与以往方法相比，本方案有效平衡了计算效率与性能，结合传统方法与大模型，在多个主流OOS检测基准上取得领先结果，包括从真实部署系统中获取的实测数据。
原文摘要 · Abstract (English)
Out-of-scope (OOS) intent detection is a critical challenge in task-oriented dialogue systems (TODS), as it ensures robustness to unseen and ambiguous queries. In this work, we propose a novel but simple modular framework that combines uncertainty modeling with fine-tuned large language models (LLMs) for efficient and accurate OOS detection. The first step applies uncertainty estimation to the output of an in-scope intent detection classifier, which is currently deployed in a real-world TODS handling tens of thousands of user interactions daily. The second step then leverages an emerging LLM-based approach, where a fine-tuned LLM is triggered to make a final decision on instances with high uncertainty. Unlike prior approaches, our method effectively balances computational efficiency and performance, combining traditional approaches with LLMs and yielding state-of-the-art results on key OOS detection benchmarks, including real-world OOS data acquired from a deployed TODS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。