按重要性合并视觉令牌，提升生成质量与效率。
Importance-Based Token Merging for Efficient Image and Video Generation
- 基于重要性评分选择关键令牌保留，避免信息丢失
- 在多类模型上实现更高图像/视频质量，尤其细节更清晰
- 适合追求高效高质生成的视觉算法研究者
令牌合并可通过仅处理相似令牌组一次并共享结果来有效加速各类视觉系统。然而，现有令牌分组方法往往随意且随机，忽视样本实际内容。我们发现，在合并过程中保留高信息量令牌——即对语义保真度和结构细节至关重要的令牌——能显著提升生成样本质量，呈现更精细的细节与更连贯真实的生成效果。尽管该思路简单直观，却未被充分探索。为此，我们提出一种基于重要性的令牌合并方法，通过利用扩散模型中现成的重要度评分（如无分类器指导）优先分配计算资源给最核心的令牌。实验表明，该方法在多个应用场景中均显著优于基线方法，涵盖文本到图像生成、多视角图像生成及视频生成，适配Stable Diffusion、Zero123++、AnimateDiff、PixArt-α等多种模型架构。
原文摘要 · Abstract (English)
Token merging can effectively accelerate various vision systems by processing groups of similar tokens only once and sharing the results across them. However, existing token grouping methods are often ad hoc and random, disregarding the actual content of the samples. We show that preserving high-information tokens during merging - those essential for semantic fidelity and structural details - significantly improves sample quality, producing finer details and more coherent, realistic generations. Despite being simple and intuitive, this approach remains underexplored. To do so, we propose an importance-based token merging method that prioritizes the most critical tokens in computational resource allocation, leveraging readily available importance scores, such as those from classifier-free guidance in diffusion models. Experiments show that our approach significantly outperforms baseline methods across multiple applications, including text-to-image synthesis, multi-view image generation, and video generation with various model architectures such as Stable Diffusion, Zero123++, AnimateDiff, or PixArt-$α$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。