用轻量框架提升故事图像生成的连贯性,仅增14.9万参数。
ReCap: Lightweight Referential Grounding for Coherent Story Visualization

- 用代词作为视觉锚点,仅在指代时激活跨帧一致性机制。
- 在两个基准上分别提升2.63%和5.65%的字符准确率,超越当前最优。
- 训练时通过语义漂移修正保持角色外观稳定,推理零开销。
故事可视化旨在生成一系列图像以忠实呈现文本叙事,保持角色身份、空间布局与风格一致。传统方法依赖显式记忆库或辅助语言模型,导致参数量与推理开销剧增。本文提出ReCap,一种轻量级一致性框架,不修改基础扩散模型即可提升角色稳定性与视觉保真度。其核心(CORE)模块将代词视为视觉锚点,仅在角色被代词提及时激活,并基于前一帧条件传播视觉身份。该设计避免了无条件跨帧条件化,仅增加14.9万参数。为增强身份稳定性,引入训练阶段的语义漂移校正(SemDrift),当文本模糊或存在指代时,通过对齐预训练DINOv3视觉嵌入,强制语义一致性,且推理阶段无需额外计算。ReCap在FlintstonesSV与PororoSV两个主流基准上分别超越先前最优的StoryGPT-V 2.63%与5.65%,建立新基准。此外,还将故事可视化拓展至真人电影衍生的叙事,验证其在非卡通场景下的适用性。
原文摘要 · Abstract (English)
Story Visualization aims to generate a sequence of images that faithfully depicts a textual narrative that preserve character identity, spatial configuration, and stylistic coherence as the narratives unfold. Maintaining such cross-frame consistency has traditionally relied on explicit memory banks, architectural expansion, or auxiliary language models, resulting in substantial parameter growth and inference overhead. We introduce ReCap, a lightweight consistency framework that improves character stability and visual fidelity without modifying the base diffusion backbone. ReCap's CORE (COnditional frame REferencing) module treats anaphors, in our case pronouns, as visual anchors, activating only when characters are referred to by a pronoun and conditioning on the preceding frame to propagate visual identity. This selective design avoids unconditional cross-frame conditioning and introduces only 149K additional parameters, a fraction of the cost of memory-bank and LLM-augmented approaches. To further stabilize identity, we incorporate SemDrift (Guided Semantic Drift Correction) applied only during training. When text is vague or referential, the denoiser lacks a visual anchor for identity-defining attributes, causing character appearance to drift across frames, SemDrift corrects this by aligning denoiser representations with pretrained DINOv3 visual embeddings, enforcing semantic identity stability at zero inference cost. ReCap outperforms previous state-of-the-art, StoryGPT-V, on the two main benchmarks for story visualization by 2.63% Character-Accuracy on FlintstonesSV and by 5.65% on PororoSV, establishing a new state-of-the-art character consistency on both benchmarks. Furthermore, we extend story visualization to human-centric narratives derived from real films, demonstrating the capability of ReCap beyond stylized cartoon domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。