用视频模型生成漫画，让角色位置和风格更一致
DreamingComics: A Story Visualization Pipeline via Subject and Layout Customized Generation using Video Models
- 基于区域感知的位置编码控制角色布局
- 角色一致性提升29.2%，风格相似度提高36.2%
- 支持自然语言生成漫画布局，适合内容创作者
当前故事可视化方法主要依赖文本定位角色，难以保持艺术风格一致性。为此，我们提出DreamingComics，一个基于布局感知的故事可视化框架。该框架基于预训练的视频扩散-Transformer（DiT）模型，利用其时空先验增强身份与风格一致性。针对布局控制，提出RegionRoPE——一种基于目标布局重索引嵌入的区域感知位置编码；同时引入掩码条件损失，进一步约束每个主体的视觉特征位于指定区域。为从自然语言脚本推断布局，集成基于LLM的布局生成器，可生成漫画风格布局，实现灵活可控的布局条件输入。全面评估显示，相比先前方法，角色一致性提升29.2%，风格相似度提高36.2%，空间准确性高。项目主页：https://yj7082126.github.io/dreamingcomics/
原文摘要 · Abstract (English)
Current story visualization methods tend to position subjects solely by text and face challenges in maintaining artistic consistency. To address these limitations, we introduce DreamingComics, a layout-aware story visualization framework. We build upon a pretrained video diffusion-transformer (DiT) model, leveraging its spatiotemporal priors to enhance identity and style consistency. For layout-based position control, we propose RegionalRoPE, a region-aware positional encoding scheme that re-indexes embeddings based on the target layout. Additionally, we introduce a masked condition loss to further constrain each subject's visual features to their designated region. To infer layouts from natural language scripts, we integrate an LLM-based layout generator trained to produce comic-style layouts, enabling flexible and controllable layout conditioning. We present a comprehensive evaluation of our approach, showing a 29.2% increase in character consistency and a 36.2% increase in style similarity compared to previous methods, while displaying high spatial accuracy. Our project page is available at https://yj7082126.github.io/dreamingcomics/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。