提升公共服务对话系统推荐可信度,关键在优化检索质量。
TRACE: Trustworthy Retrieval-Augmented Conversational Engine

- 通过结构化与语义约束解析用户查询,增强检索精准度
- 检索质量提升后,用户约束满足率显著提高,幻觉推荐减少
- 对小模型也有效,适合依赖可靠数据的政务类应用
公共服务聊天机器人需基于公开服务目录提供推荐,同时遵守用户的显式约束。然而实际中服务目录噪声大、不一致,通用大语言模型常生成不可靠推荐,引用未经验证的网络信息。本文研究检索质量对约束感知推荐的影响,提出 TRACE(可信检索增强型对话引擎),利用双数据表示架构将用户查询解析为结构化和语义约束,用于下游检索。基于一个州级食品援助目录和合成查询基准,评估了多种知识表示变体(含/不含知识图谱)。实验对比多个开源及专有大模型,结果表明:强化检索可显著提升用户约束满足率,降低幻觉推荐;不同模型间性能差异缩小,系统对模型规模的敏感性下降。这表明检索质量是构建稳健公共服务对话系统的关键。
原文摘要 · Abstract (English)
Public service chatbots are expected to deliver recommendations from an underlying public service directory, while also making sure that the recommendations respect explicit user constraints. In practice, public service directories are noisy and inconsistent, and general-purpose large language model (LLM) or AI-based chatbots frequently generate unreliable recommendations, citing unverified sources from the web. We investigate the impact of retrieval quality on constraint-aware recommendation in public service conversational systems built over noisy and heterogeneous service directories. We propose TRACE (Trustworthy Retrieval-Augmented Conversational Engine), a retrieval-based, constraint-aware framework that parses input user queries into structural and semantic constraints for downstream retrieval, with the help of a dual data representation schema. Using a curated statewide pantry directory and a synthetic query benchmark, we evaluate multiple knowledge-representation variants with and without knowledge graphs (KGs). We experiment with several open-source LLMs and a proprietary model, showing that strengthening retrieval substantially improves user constraint satisfaction while reducing hallucinated recommendations. Performance differences across LLMs narrowed in our experiments as retrieval quality improved, making results less sensitive to model size. These findings suggest that the quality of retrieval is key for robust public service conversational systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。