让图像像文字一样嵌入指令,实现更精准的多图生成
Images in Sentences: Scaling Interleaved Instructions for Unified Visual Generation

- 将图像作为语义位置的原生词汇,直接融入文本指令中
- 在1500万合成样本上训练,复杂指令下生成一致性提升显著
- 适合需要高精度图文对齐与创意视觉编辑的研究者
尽管多模态语言模型已能根据多图指令生成图像,但现有方法在复杂交错指令下性能下降。其根源在于图像与文本在结构上的分离,导致模型需处理长距离依赖以匹配描述与视觉目标。为此,我们提出INSET(Images in Sentences),一种将图像作为原生词汇无缝嵌入文本指令的统一生成模型。通过将视觉特征置于对应语义位置,INSET利用Transformer的上下文局部性实现精确对象绑定,将图像视为密集、富有表现力的语言标记。此外,我们构建了一个可扩展的数据引擎,从标准图像和视频数据集合成1500万条高质量交错样本,利用视觉语言模型(VLMs)和大语言模型(LLMs)生成丰富、长时序序列。在InterleaveBench上的评估表明,INSET在多图一致性和文本对齐方面显著优于当前最佳方法,且输入复杂度越高,性能差距越明显。该方法还可自然延伸至多模态图像编辑,将视觉内容作为指令一部分,支持高度表达性与创造性的视觉操作。
原文摘要 · Abstract (English)
While recent advancements in multimodal language models have enabled image generation from expressive multi-image instructions, existing methods struggle to maintain performance under complex interleaved instructions. This limitation stems from the structural separation of images and text in current paradigms, which forces models to bridge difficult long-range dependencies to match descriptions with visual targets. To address these challenges, we propose \texttt{I}mages i\texttt{N} \texttt{SE}n\texttt{T}ences (\textit{a.k.a}, INSET), a unified generation model that seamlessly embeds images as native vocabulary within textual instructions. By positioning visual features directly at their corresponding semantic slots, INSET leverages the contextual locality of transformers for precise object binding, effectively treating images as dense, expressive language tokens. Furthermore, we introduce a scalable data engine that synthesizes 15M high-quality interleaved samples from standard image and video datasets, utilizing VLMs and LLMs to construct rich, long-horizon sequences. Evaluation results on InterleaveBench demonstrate that INSET significantly outperforms state-of-the-art methods in multi-image consistency and text alignment, with performance gaps widening as input complexity increases. Beyond standard generation, our approach inherently extends to multimodal image editing, integrating visual content as part of the instruction to facilitate highly expressive and creative visual manipulations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。