动态合并低重要性token，实现高效且兼容自回归生成的文本/视觉生成。
QuickMerge++: Fast Token Merging with Autoregressive Prior
- 根据注意力幅值动态选择关键token，结合熵估算预算。
- 在多模态任务中减少70%以上token数，性能持平甚至超越基线。
- 适合需要降低推理开销的生成模型部署场景。
随着生成模型在语言、视觉和视频领域处理更大输入规模，基于token的计算成本已成为主要瓶颈。尽管已有研究指出仅部分token对下游预测有显著影响，但多数现有方法为静态设定、仅限特定模态或不兼容自回归生成。本文提出QuickMerge，一种轻量级的token合并框架，用于高效预测下一个token。该方法基于注意力幅值动态选取较少的候选token，由基于熵的预算估计器控制。为保持自回归一致性，引入一个在合并后的序列上训练的轻量Transformer先验模型。通过融合语义显著性评估、灵活的token预算与自回归对齐，QuickMerge实现了用更少token完成准确生成。我们在多模态场景下评估该方法，证明其在计算-精度权衡上持续优化。具体而言，QuickMerge大幅减少token数量，同时性能达到甚至超过学习型tokenizer和固定块基线。
原文摘要 · Abstract (English)
As generative models scale to larger inputs across language, vision, and video domains, the cost of token-level computation has become a key bottleneck. While prior work suggests that only a subset of tokens significantly influence downstream predictions, most token selection methods are static, modality-specific, or incompatible with autoregressive generation. In this paper, we propose QuickMerge, a lightweight token merging framework designed for efficient next-token prediction. QuickMerge dynamically selects a reduced number of tokens based on attention norm magnitude, guided by an entropy-based budget estimator. To preserve autoregressive compatibility, we introduce a lightweight transformer prior trained over the merged token sequence. By combining semantic salience estimation, flexible token budgets, and AR alignment, QuickMerge enables accurate generation with fewer tokens. We evaluate QuickMerge across multi-modality domains, demonstrating consistent improvements in compute-accuracy tradeoffs. Specifically, QuickMerge reduces token counts sustantially while matching as well as exceeding the performance of learned tokenizers and fixed-patch baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。