提出无监督协作压缩框架,让多模态大模型在降耗同时保持音视频理解性能
OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Models

- 先对齐跨模态片段边界,再协同压缩音视频标记
- 44%标记保留率下仍达46.7%准确率,接近全标记基线
- 无需训练,适合部署资源受限的多模态系统
多模态大语言模型(Omni-LLMs)在音视频理解方面表现强劲,但长视频流和密集音频序列带来的高推理成本限制了其实际应用。现有压缩方法通常依赖固定或原始压缩单元,易破坏跨模态对应关系与互补信息,难以在提升效率的同时稳定保持性能。为此,我们提出OmniRefine,一种无需训练的两阶段音视频标记压缩框架。首先,通过帧-音频相似性与动态规划实现跨模态对齐的分块精炼,重构压缩单元边界;其次,在每个精炼单元内实施模态感知的协同压缩,降低冗余同时保留关键证据。大量实验表明,OmniRefine在效率-性能权衡上优于强基线,且在低压缩比下表现稳定。在WorldSense数据集上,44%标记保留率下仍达46.7%准确率,几乎匹配全标记基线。代码与接口将公开,以促进后续研究。
原文摘要 · Abstract (English)
Omnimodal large language models (Omni-LLMs) show strong capability in audio-video understanding, but their practical deployment remains limited by high inference cost of long video streams and dense audio sequences. Despite recent progress, existing compression methods for Omni-LLMs typically rely on fixed or native compression units, which can disrupt cross-modal correspondence and the complementary information required for audio-video reasoning, making it difficult to improve inference efficiency while stably preserving performance. To address this, we propose OmniRefine, a training-free two-stage framework for efficient audio-visual token compression in Omni-LLMs. First, Correspondence-Preserving Chunk Refinement refines native chunk boundaries into cross-modally aligned compression units through frame-audio similarity and dynamic programming. Second, Modality-Aware Cooperative Compression jointly compresses video and audio tokens within each refined unit to reduce redundancy while preserving critical evidence. Extensive experiments show that OmniRefine achieves a better efficiency-performance trade-off than strong baselines and maintains stable performance under lower compression ratios. On WorldSense, it still reaches 46.7% accuracy at a 44% token retention ratio, nearly matching the full-token baseline. The code and interface will be released to facilitate further research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。