提出检测-解释-升级框架,让大模型对话更稳定可靠。
Detect, Explain, Escalate: Sustainable Dialogue Breakdown Management for LLM Agents
- 用80亿参数小模型实时检测对话故障并生成解释。
- 在多个数据集上提升准确率7%,推理成本降低54%。
- 适合需要高可靠性、低耗能的对话系统部署场景。
大型语言模型在对话应用中表现出强大能力,但易发生对话中断,影响部署可靠性和用户信任。本文提出“检测-解释-升级”框架,实现资源高效的对话中断管理。方法包括:(1) 微调一个80亿参数的小模型,结合教师生成的推理轨迹,作为实时中断检测与解释器,在英语和日语对话中表现稳健,并在BETOLD数据集上比基线提升7%准确率;(2) 使用前沿大模型结合少样本、思维链和类比推理等高级提示策略进行高保真中断评估,并构建“升级”架构,仅在必要时调用大模型,显著降低计算开销。该方案在DBDC5上达到顶尖性能,优于专用分类器,并缩小与大型专有模型的差距。整体监测-升级流程将推理成本降低54%,为关键领域提供低成本、可解释的鲁棒对话解决方案。代码与模型已公开。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated substantial capabilities in conversational AI applications, yet their susceptibility to dialogue breakdowns poses significant challenges to deployment reliability and user trust. This paper introduces a "Detect, Explain, Escalate" framework to manage dialogue breakdowns in LLM-powered agents, emphasizing resource-efficient operation. Our approach integrates two key strategies: (1) We fine-tune a compact 8B-parameter model, augmented with teacher-generated reasoning traces, which serves as an efficient real-time breakdown detector and explainer. This model demonstrates robust classification and calibration on English and Japanese dialogues, and generalizes to the BETOLD dataset, improving accuracy by 7% over its baseline. (2) We systematically evaluate frontier LLMs using advanced prompting (few-shot, chain-of-thought, analogical reasoning) for high-fidelity breakdown assessment. These are integrated into an "escalation" architecture where our efficient detector defers to larger models only when necessary, substantially reducing operational costs and computational overhead. Our fine-tuned model and prompting strategies achieve state-of-the-art performance on DBDC5 and strong results on BETOLD, outperforming specialized classifiers on DBDC5 and narrowing the performance gap to larger proprietary models. The proposed monitor-escalate pipeline reduces inference costs by 54%, providing a cost-effective and interpretable solution for robust conversational AI in high-impact domains. Code and models are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。