让AI看图像序列时能理解时间顺序，提升连贯描述能力。
ImageChain: Advancing Sequential Image-to-Text Reasoning in Multimodal Large Language Models
- 将图像序列转为多轮对话形式，显式建模时间依赖关系。
- 在下一场景描述任务中，SimRate指标提升3.7%至19%。
- 适用于漫画、机器人等跨领域场景，零样本表现稳健。
序列图像推理仍是多模态大模型的挑战。尽管现有模型在预训练中引入多图像数据，仍难以识别序列结构，常将图像视为独立个体。本文提出ImageChain框架，通过将图像与对应文本描述交替排列，构建受控对话形式，显式捕捉视觉序列的时间依赖与叙事进展。该方法以生成下一场景的上下文感知描述为目标，优化模型对时序信息的理解。实验表明，该方法在下一场景描述任务中，使SimRate（衡量与人工标注真实值语义相似度的指标）平均提升3.7%至19%。此外，ImageChain在漫画、机器人等跨领域任务中表现出强零样本泛化能力。大量实验证实，采用多模态多轮对话设计进行指令微调，是连接静态图像理解与时序推理的关键。
原文摘要 · Abstract (English)
Reasoning over sequences of images remains a challenge for multimodal large language models (MLLMs). While recent models incorporate multi-image data during pre-training, they still struggle to recognize sequential structures, often treating images independently. This work introduces ImageChain, a framework that enhances MLLMs with sequential reasoning capabilities over image data by modeling visual sequences as a multi-turn conversation. In ImageChain, images are interleaved with corresponding textual descriptions to form a controlled dialogue that explicitly captures temporal dependencies and narrative progression. Our method optimizes for the task of next-scene description, where the model generates a context-aware description of an upcoming scene based on preceding visual and textual cues. We demonstrate that our approach improves performance on the next-scene description task -- achieving an average improvement from 3.7% to 19% in SimRate, a metric that quantifies semantic similarity to human-annotated ground truths. Moreover, ImageChain achieves robust zero-shot out-of-domain performance in applications ranging from comics to robotics. Extensive experiments validate that instruction-tuning in a multimodal, multi-turn conversation design is key to bridging the gap between static image understanding and temporally-aware reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。