用智能删减token加速多模态大模型训练，省41%算力仍更准
ReGATE: Learning Faster and Better with Fewer Tokens in MLLMs
- 通过师生框架动态判断每段token重要性，只保留关键信息
- 训练速度提升2倍，仅用38%的token达到相同准确率
- 适合追求高效训练的AI研发者，尤其适合资源受限场景
多模态大语言模型（MLLM）的训练计算成本随处理标记数急剧上升。现有高效方法主要针对推理阶段的标记压缩或合并，对训练阶段帮助有限。本文提出ReGATE（参考引导自适应标记剔除），一种用于加速MLLM训练的自适应标记剪枝方法。ReGATE采用冻结教师模型提供逐标记引导损失，并与学生模型难度估计的指数移动平均值融合，形成自适应评分机制。该机制在前向传播中动态选择有信息量的标记，跳过冗余内容，显著降低计算开销且不改变模型结构。在三个代表性MLLM上，ReGATE在MVBench上以仅38%的标记数达到标准训练的峰值准确率，速度快达2倍；经更长周期训练后，其性能甚至超越基线，在多个多模态基准上总标记使用量减少超41%。
原文摘要 · Abstract (English)
The computational cost of training multimodal large language models (MLLMs) grows rapidly with the number of processed tokens. Existing efficiency methods mainly target inference via token reduction or merging, offering limited benefits during training. We introduce ReGATE (Reference-Guided Adaptive Token Elision), an adaptive token pruning method for accelerating MLLM training. ReGATE adopts a teacher-student framework, in which a frozen teacher LLM provides per-token guidance losses that are fused with an exponential moving average of the student's difficulty estimates. This adaptive scoring mechanism dynamically selects informative tokens while skipping redundant ones in the forward pass, substantially reducing computation without altering the model architecture. Across three representative MLLMs, ReGATE matches the peak accuracy of standard training on MVBench up to 2$\times$ faster, using only 38% of the tokens. With extended training, it even surpasses the baseline across multiple multimodal benchmarks, cutting total token usage by over 41%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。