arXiv:2505.18227cs.LGcs.AI2025-05被引 46

Token减少不只是提速,更是提升生成模型质量的关键机制。

Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality

  • 将令牌压缩从效率优化升级为生成建模的核心原则。
  • 可缓解幻觉、增强长文本连贯性与多模态对齐效果。
  • 适合关注生成质量、模型稳定性的研究者与开发者。

在Transformer架构中,令牌是通过将输入数据分割为固定长度块生成的离散单元,每个令牌映射为嵌入向量,以支持并行注意力计算并保留输入关键信息。由于Transformer自注意力机制存在二次计算复杂度,令牌减少主要被用作提升效率的策略,尤其在视觉和语言单模态任务中,有助于平衡计算成本、内存占用与推理延迟。然而,本文认为,在大规模生成模型时代,令牌减少不应仅限于效率考量。我们提出将其视为生成建模的基本原则,深刻影响模型架构与应用拓展。具体而言,它可在视觉、语言及多模态系统中实现:(i) 深化多模态融合与对齐,(ii) 缓解‘过度思考’与幻觉问题,(iii) 维持长序列输入的连贯性,(iv) 提升训练稳定性等。本文重新定义令牌减少的意义,展望未来方向,包括算法设计、强化学习引导的令牌缩减、上下文学习中的令牌优化、智能体框架构建,以及更广泛的机器学习与科学领域应用。

原文摘要 · Abstract (English)

In Transformer architectures, tokens\textemdash discrete units derived from raw data\textemdash are formed by segmenting inputs into fixed-length chunks. Each token is then mapped to an embedding, enabling parallel attention computations while preserving the input's essential information. Due to the quadratic computational complexity of transformer self-attention mechanisms, token reduction has primarily been used as an efficiency strategy. This is especially true in single vision and language domains, where it helps balance computational costs, memory usage, and inference latency. Despite these advances, this paper argues that token reduction should transcend its traditional efficiency-oriented role in the era of large generative models. Instead, we position it as a fundamental principle in generative modeling, critically influencing both model architecture and broader applications. Specifically, we contend that across vision, language, and multimodal systems, token reduction can: (i) facilitate deeper multimodal integration and alignment, (ii) mitigate "overthinking" and hallucinations, (iii) maintain coherence over long inputs, and (iv) enhance training stability, etc. We reframe token reduction as more than an efficiency measure. By doing so, we outline promising future directions, including algorithm design, reinforcement learning-guided token reduction, token optimization for in-context learning, agentic framework design, and broader ML and scientific domains.

生成模型多模态令牌压缩效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。