Loom实现图文交替生成,提升长序列一致性与对齐度。
Loom: Diffusion-Transformer for Interleaved Generation
- 采用交错嵌入架构,交替处理文本与图像信息以支持多条件推理。
- 在文本到图文任务中平均得分提升2.6分(5分制),显著优于基线。
- 适合需要连贯图文生成的场景,如教学视频、创意合成等。
交错式文本-图像生成旨在单个序列中联合生成连贯的视觉帧与对齐的文本描述,适用于风格迁移、组合生成和程序化教程等任务。我们提出Loom,一种统一的扩散-变压器框架,用于交错式文本-图像生成。Loom通过全参数微调扩展Bagel统一模型,并采用交错架构,交替排列文本与视觉嵌入,支持多条件推理与顺序规划。语言规划策略将用户指令分解为逐步提示与帧嵌入,引导时序一致的合成。对于每帧,Loom仅依赖少量采样前序帧及全局文本上下文,而非拼接全部历史,实现可控且高效的长时程生成。在风格迁移、组合生成和教程类流程中,Loom展现出更优的组合性、时序一致性和图文对齐性。实验表明,Loom在开放源代码基线Anole基础上,于文本到交错生成任务中平均提升2.6分(5分制),在时序与语义指标上均有显著优势。我们还构建了一个5万条的交错教程数据集,并展示了其在统一模型与扩散编辑基线上的显著改进。
原文摘要 · Abstract (English)
Interleaved text-image generation aims to jointly produce coherent visual frames and aligned textual descriptions within a single sequence, enabling tasks such as style transfer, compositional synthesis, and procedural tutorials. We present Loom, a unified diffusion-transformer framework for interleaved text-image generation. Loom extends the Bagel unified model via full-parameter fine-tuning and an interleaved architecture that alternates textual and visual embeddings for multi-condition reasoning and sequential planning. A language planning strategy first decomposes a user instruction into stepwise prompts and frame embeddings, which guide temporally consistent synthesis. For each frame, Loom conditions on a small set of sampled prior frames together with the global textual context, rather than concatenating all history, yielding controllable and efficient long-horizon generation. Across style transfer, compositional generation, and tutorial-like procedures, Loom delivers superior compositionality, temporal coherence, and text-image alignment. Experiments demonstrate that Loom substantially outperforms the open-source baseline Anole, achieving an average gain of 2.6 points (on a 5-point scale) across temporal and semantic metrics in text-to-interleaved tasks. We also curate a 50K interleaved tutorial dataset and demonstrate strong improvements over unified and diffusion editing baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。