仅用语言模型识别对话中指代对象，发现语境足够支撑部分判断。
Detecting Referring Expressions in Visually Grounded Dialogue with Autoregressive Language Models
- 用自回归语言模型通过预测下一个词定位指代片段边界。
- 小数据+轻量微调下仍能实现有效指代检测，表明语言上下文作用显著。
- 适合研究多模态任务中单模态方法的边界与局限性。
本文探索了仅使用文本的自回归语言建模方法，从视觉对话中提取指代表达。核心目标是考察仅依靠语言上下文能否有效识别具有视觉对应物的提及内容。为此，我们对预训练的大语言模型进行适配，通过下一词预测来划分对话文本中的指代片段边界，完成粗粒度标注。实验结果表明，即使采用中等规模的LLM、较小的数据集以及参数高效微调，纯文本方法依然有效，凸显了语言上下文在该任务中的关键作用。然而，我们指出该任务本质上是多模态问题，且单模态方法存在根本性局限。
原文摘要 · Abstract (English)
In this paper, we explore the use of a text-only, autoregressive language modeling approach for the extraction of referring expressions from visually grounded dialogue. More specifically, the aim is to investigate the extent to which the linguistic context alone can inform the detection of mentions that have a (visually perceivable) referent in the visual context of the conversation. To this end, we adapt a pretrained large language model (LLM) to perform a relatively course-grained annotation of mention spans in unfolding conversations by demarcating mention span boundaries in text via next-token prediction. Our findings indicate that even when using a moderately sized LLM, relatively small datasets, and parameter-efficient fine-tuning, a text-only approach can be effective, highlighting the relative importance of the linguistic context for this task. Nevertheless, we argue that the task represents an inherently multimodal problem and discuss limitations fundamental to unimodal approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。