构建多模态烹饪指令数据集，提升模型对隐含动作要素的推理能力。
Predicting Implicit Arguments in Procedural Video Instructions
- 构建Implicit-VidSRL数据集，需结合视觉上下文推断隐含论元
- 现有模型在隐含'什么'和'何处/用具'角色预测上准确率不足
- 提出iSRL-Qwen2-VL模型，相对GPT-4o提升17%和14.7%的F1分数
程序性文本有助于AI理解上下文与动作序列。将其转化为语义角色标注（SRL）可识别动词-论元结构，如{动词, 什么, 何处/用具}。烹饪类程序指令高度省略，例如（i）将黄瓜放入碗中，（ii）加入切好的番茄——第二步的‘何处’论元需从上下文推断，即前一步放置黄瓜的位置。以往的SRL基准常忽略隐含论元，导致理解不完整。为此，我们提出Implicit-VidSRL数据集，要求模型从多模态烹饪流程中推断隐含与显式论元。该数据集评估多模态模型的上下文推理能力，需追踪视觉变化中的实体。我们测试了近期多模态大模型，发现其在给定动词时仍难以预测‘什么’和‘何处/用具’的隐含角色。为此，我们提出iSRL-Qwen2-VL模型，在what-implicit和where/with-implicit的语义角色上，相比GPT-4o分别实现17%和14.7%的相对F1分数提升。
原文摘要 · Abstract (English)
Procedural texts help AI enhance reasoning about context and action sequences. Transforming these into Semantic Role Labeling (SRL) improves understanding of individual steps by identifying predicate-argument structure like {verb,what,where/with}. Procedural instructions are highly elliptic, for instance, (i) add cucumber to the bowl and (ii) add sliced tomatoes, the second step's where argument is inferred from the context, referring to where the cucumber was placed. Prior SRL benchmarks often miss implicit arguments, leading to incomplete understanding. To address this, we introduce Implicit-VidSRL, a dataset that necessitates inferring implicit and explicit arguments from contextual information in multimodal cooking procedures. Our proposed dataset benchmarks multimodal models' contextual reasoning, requiring entity tracking through visual changes in recipes. We study recent multimodal LLMs and reveal that they struggle to predict implicit arguments of what and where/with from multi-modal procedural data given the verb. Lastly, we propose iSRL-Qwen2-VL, which achieves a 17% relative improvement in F1-score for what-implicit and a 14.7% for where/with-implicit semantic roles over GPT-4o.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。