对比多种方法,发现传统模型在常见输入上更高效,神经模型更适合未知输入。
Chat-Ghosting: A Comparative Study of Methods for Auto-Completion in Dialog Systems
- 用字典树和n-gram等传统方法结合对话上下文提升补全准确率
- 统计模型在已见前缀上性能优于深度学习模型,且推理更快
- 新提出的动态停止策略可加速生成,适合实际对话系统部署
Ghosting(输入补全)是现代搜索与聊天系统中预测用户意图输入的重要功能,能显著改善打字慢、语言能力弱或有残障用户的体验。尽管基于聊天的系统(如ChatGPT、Copilot)广泛应用此功能,但该问题在自然语言处理领域仍缺乏系统研究。本文使用四个公开对话数据集(两个真人对话:DailyDialog、DSTC7-Ubuntu;两个真人-机器人对话:Open Assistant、ShareGPT),对比了基于字典树、n-gram及深度学习的方法,包含是否使用对话上下文的设置。提出一种基于熵的动态提前终止策略。结果表明:对于已见过的输入前缀,统计模型和字典树在性能与效率上均优于神经模型;对于未见过的查询,T5和Phi-2等神经模型表现更优。引入对话上下文可显著提升补全质量,尤其在Open Assistant和ShareGPT数据集上。代码与数据已开源。
原文摘要 · Abstract (English)
Ghosting, the ability to predict a user's intended text input for inline query auto-completion, is an invaluable feature for modern search engines and chat interfaces, greatly enhancing user experience. By suggesting completions to incomplete queries (or prefixes), ghosting aids users with slow typing speeds, disabilities, or limited language proficiency. Ghosting is a challenging problem and has become more important with the ubiquitousness of chat-based systems like ChatGPT, Copilot, etc. Despite the increasing prominence of chat-based systems utilizing ghosting, this challenging problem of Chat-Ghosting has received little attention from the NLP/ML research community. There is a lack of standardized benchmarks and relative performance analysis of deep learning and non-deep learning methods. We address this through an open and thorough study of this problem using four publicly available dialog datasets: two human-human (DailyDialog and DSTC7-Ubuntu) and two human-bot (Open Assistant and ShareGPT). We experiment with various existing query auto-completion methods (using tries), n-gram methods and deep learning methods, with and without dialog context. We also propose a novel entropy-based dynamic early stopping strategy. Our analysis finds that statistical n-gram models and tries outperform deep learning based models in terms of both model performance and inference efficiency for seen prefixes. For unseen queries, neural models like T5 and Phi-2 lead to better results. Adding conversational context leads to significant improvements in ghosting quality, especially for Open-Assistant and ShareGPT. We make code and data publicly available
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。