用双类知识增强对话生成,让AI更懂用户意图。
Dual Knowledge-Enhanced Two-Stage Reasoner for Multimodal Dialog Systems
- 双阶段推理:先选知识类型,再分离意图与回复
- 在公开数据集上优于基线模型,提升响应相关性
- 适合做多模态任务型对话系统的开发者
文本响应生成对多模态任务导向对话系统至关重要,需基于多模态上下文生成恰当回应。现有方法存在忽视非结构化评论知识和未充分使用大语言模型(LLMs)的问题。为此,我们提出双知识增强两阶段推理器(DK2R),通过融合结构化属性与非结构化评论知识,并利用LLMs提升生成质量。该方法面临两大挑战:动态知识类型选择与意图-响应解耦。DK2R首先从外部知识库提取两类知识;随后用LLM评估各知识类型的效用,生成初步探针响应;最后分别提炼意图相关关键线索,作为辅助信号优化响应生成。在公开数据集上的大量实验验证了其优越性。代码与参数已开源。
原文摘要 · Abstract (English)
Textual response generation is pivotal for multimodal \mbox{task-oriented} dialog systems, which aims to generate proper textual responses based on the multimodal context. While existing efforts have demonstrated remarkable progress, there still exist the following limitations: 1) \textit{neglect of unstructured review knowledge} and 2) \textit{underutilization of large language models (LLMs)}. Inspired by this, we aim to fully utilize dual knowledge (\textit{i.e., } structured attribute and unstructured review knowledge) with LLMs to promote textual response generation in multimodal task-oriented dialog systems. However, this task is non-trivial due to two key challenges: 1) \textit{dynamic knowledge type selection} and 2) \textit{intention-response decoupling}. To address these challenges, we propose a novel dual knowledge-enhanced two-stage reasoner by adapting LLMs for multimodal dialog systems (named DK2R). To be specific, DK2R first extracts both structured attribute and unstructured review knowledge from external knowledge base given the dialog context. Thereafter, DK2R uses an LLM to evaluate each knowledge type's utility by analyzing LLM-generated provisional probe responses. Moreover, DK2R separately summarizes the intention-oriented key clues via dedicated reasoning, which are further used as auxiliary signals to enhance LLM-based textual response generation. Extensive experiments conducted on a public dataset verify the superiority of DK2R. We have released the codes and parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。