构建首个融合视觉与显式推理的语音理解基准,提升真实场景下对话系统表现。
Introducing Visual Scenes and Reasoning: A More Realistic Benchmark for Spoken Language Understanding

- 用GPT-4o生成用户环境图像,替代理想化上下文向量
- 引入人工验证的解释文本,实现标签预测与推理分离
- 提出两步式指令模板,缓解推理对预测的偏差干扰
语音理解(SLU)包含意图识别(ID)和槽位填充(SF)两个子任务。为提升实际部署能力,现有研究引入上下文感知(CA)、用户画像(UP)和知识图谱(KG)以应对语义模糊问题。然而,现有数据集仍无法充分反映真实场景:(1) 上下文感知仅使用独热编码表示,过于理想化;(2) 模型仅关注标签预测,忽略可解释的推理过程。为此,本文提出VRSLU数据集,融合视觉图像与显式推理。通过GPT-4o和FLUX.1-dev生成反映用户环境与状态的图像,并经人工验证确保质量;利用GPT-4o生成预测标签的解释,再由人工修正以保证准确与连贯。此外,设计了双阶段指令模板LR-Instruct:先预测标签,再生成对应推理,有效降低推理偏见对预测的影响。实验验证了视觉信息的有效性及显式推理在提升性能与可解释性方面的潜力。
原文摘要 · Abstract (English)
Spoken Language Understanding (SLU) consists of two sub-tasks: intent detection (ID) and slot filling (SF). Given its broad range of real-world applications, enhancing SLU for practical deployment is increasingly critical. Profile-based SLU addresses ambiguous user utterances by incorporating context awareness (CA), user profiles (UP), and knowledge graphs (KG) to support disambiguation, thereby advancing SLU research toward real-world applicability. However, existing SLU datasets still fall short in representing real-world scenarios. Specifically, (1) CA uses one-hot vectors for representation, which is overly idealized, and (2) models typically focuses solely on predicting intents and slot labels, neglecting the reasoning process that could enhance performance and interpretability. To overcome these limitations, we introduce VRSLU, a novel SLU dataset that integrates both Visual images and explicit Reasoning. For over-idealized CA, we use GPT-4o and FLUX.1-dev to generate images reflecting users' environments and statuses, followed by human verification to ensure quality. For reasoning, GPT-4o is employed to generate explanations for predicted labels, which are then refined by human annotators to ensure accuracy and coherence. Additionally, we propose an instructional template, LR-Instruct, which first predicts labels and then generates corresponding reasoning. This two-step approach helps mitigate the influence of reasoning bias on label prediction. Experimental results confirm the effectiveness of incorporating visual information and highlight the promise of explicit reasoning in advancing SLU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。