让多模态模型高效记忆,保持视听连贯性的同时压缩参数。
Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models

- 用低秩适配器+分组优化策略,把多模态记忆转为固定预算参数。
- 30%参数预算下,准确率比最强基线高8-12%,压缩到75%仍稳定。
- 适合需要快速响应的长视频问答场景,显著降低推理延迟。
多模态语言模型(OLMs)实现统一的音视频理解,但处理长时序联合标记序列导致推理计算开销巨大。现有令牌压缩方法常孤立压缩模态,破坏跨模态时间锚点,影响连贯推理。本文提出Omni2LoRA,一种两阶段框架,通过保持连贯性的上下文蒸馏实现高效参数化记忆压缩,彻底避开令牌瓶颈。首先,一个Perceiver超网络在单次前向传播中,将冻结的OLM中间表示编码为全秩低秩适配(LoRA)适配器。为防止参数量随记录长度线性增长,采用组相对策略优化(GRPO)优化离散秩分配策略,利用模态消融反事实奖励显式惩罚视听连贯性损失,迫使模型将固定次线性秩预算分配给协同的跨模态锚点而非孤立视觉特征。在三个多模态骨干模型上,Omni2LoRA在30%秩预算下优于直接全上下文推理及强基线(OmniZip、OMAC、O-MARC),在四个音视频问答基准上平均准确率提升8-12%,压缩比达75%时仍保持稳定,而令牌剪枝方法性能急剧下降。通过将多模态记忆转化为固定预算、可复用的参数状态,本方法使回答时多模态令牌负载归零,相比全上下文推理将每查询首次生成时间(TTFT)降低最高12倍,经少量查询后延迟降至0.5秒以下。
原文摘要 · Abstract (English)
Omnimodal language models (OLMs) enable unified audio-visual understanding, but processing long joint token sequences makes inference computationally prohibitive. While recent token compression methods attempt to alleviate this burden, compressing modalities in isolation often destroys the temporal cross-modal anchors necessary for coherent reasoning. We introduce Omni2LoRA, a two-stage framework for efficient parametric memory compression via coherence-preserving context distillation that bypasses the token bottleneck entirely. First, a Perceiver hypernetwork processes intermediate representations from a frozen OLM to encode the multimodal context into a full-rank Low-Rank Adaptation (LoRA) adapter in a single forward pass. To prevent the resulting parameter footprint from scaling linearly with recording length, we optimize a discrete rank allocation policy via Group Relative Policy Optimization (GRPO) that uses a modality-ablated counterfactual reward to explicitly penalize the loss of audio-visual coherence, forcing the model to allocate its fixed sub-linear rank budget to synergistic cross-modal anchors rather than isolated visual features. Across three omnimodal backbones, Omni2LoRA operating at a 30% rank budget outperforms direct full-context inference and strong token-compression baselines (OmniZip, OMAC, O-MARC) on four audio-visual question answering benchmarks, improving average accuracy by 8-12% over the strongest baseline and remaining stable under compression ratios as tight as 75%, where token-pruning methods degrade sharply. By converting multimodal memory into a fixed-budget, reusable parameter state, our method drives answer-time multimodal-token load to zero, cutting per-query Time to First Token (TTFT) by up to 12x relative to full-context inference and amortizing to under 0.5s after a handful of queries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。