用声音和文字压缩视频令牌，让多模态大模型推理更快更省。
Out of Sight, Still in Mind: Token Compression for Omni-LLMs
- 将音视频对齐到同一空间，只保留独特信息的视觉令牌
- 用文字描述替代物体级视觉令牌，减少54%输入令牌数
- 无需训练，适配多模态大模型，提升推理效率
本文旨在降低多模态大语言模型（Omni-LLMs）推理时的输入令牌开销。这类模型需联合处理音频、视频与文本，但三者成本极不平衡：视觉令牌占绝大多数且高度冗余。本文提出ReMo，一种免训练框架，通过跨模态信息重分配压缩视觉令牌：仅保留其他模态无法覆盖的信息。具体方法包括：(i) 将音频与视频映射至统一嵌入空间，移除已被音频或其它视觉令牌解释的内容；(ii) 以紧凑文本代理替代物体级视觉令牌，即用简短描述每个物体及其位置，显著减少令牌数量。在Qwen2.5-Omni两个模型规模上，ReMo移除54%输入令牌，准确率无损失，甚至略超全令牌模型，分别达到其平均准确率的101.2%和101.3%（五项音视频基准测试均值）。
原文摘要 · Abstract (English)
The goal of this paper is to reduce the input token cost of Omni-modal large language models (Omni-LLMs) at inference time. Omni-LLMs reason jointly over audio, video and text, but the cost of the three streams is highly unbalanced: visual tokens account for the vast majority of the input, and are highly redundant. In this paper, we propose ReMo, a training-free framework that compresses visual tokens by redistributing their information across modalities: a visual token is kept only if its information appears nowhere else. ReMo achieves this in two ways: (i) it aligns audio and video in a common embedding space, and removes visual tokens already explained by the audio or by other visual tokens; and (ii) it replaces object-level visual tokens with compact text proxies, short descriptions of each object and its location, conveying the same content in far fewer tokens. On Qwen2.5-Omni at two model scales, ReMo removes 54% of the input tokens with no loss in accuracy. Indeed, it slightly exceeds the full-token model, reaching 101.2% and 101.3% of its average accuracy over five audio-visual benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。