针对音视频大模型长视频推理的内存瓶颈，提出分模态压缩新方法。
OmniMem: Perturbation-aware Memory Compression for Streaming Audio-Visual LLMs
- 按视觉与音频分别管理记忆，缓解模态间令牌不平衡问题。
- 在相同内存预算下，准确率比现有方法高2-4%，微调后提升1-2%。
- 适合需要高效处理长视频的音视频理解系统开发者使用。
音视频大语言模型在长视频理解方面潜力巨大，但其长视频推理受视频令牌和键值（KV）缓存线性增长的制约。我们提出OmniMem，一种专为音视频大模型设计的流式内存高效框架。不同于传统统一处理所有令牌的压缩方法，OmniMem采用模态感知的记忆分配策略，分别管理视觉与音频上下文，解决两模态间严重的令牌失衡问题。通过扰动感知的记忆选择机制，保留信息丰富且非冗余的KV状态，实现紧凑记忆而不损失长程理解能力。为强化实际部署条件下的压缩效果，还探索了预算感知微调，促使模型将有用信息集中于保留的记忆中。在VideoMME Long、LVBench和LVOmniBench上使用video-SALMONN 2+和Qwen-2.5-Omni的实验表明，OmniMem在相同内存预算下，相比强训练无关压缩基线，绝对准确率提升2-4%，微调后额外提升1-2%。
原文摘要 · Abstract (English)
Audio-visual large language models (LLMs) hold strong promise for long-form video understanding, yet their long-video inference is fundamentally limited by the linear growth of video tokens and key-value (KV) caches. We present OmniMem, a memory-efficient streaming framework designed specifically for audio-visual LLMs. Unlike existing compression methods that treat all tokens uniformly, OmniMem introduces a modality-aware memory allocation strategy that separately manages visual and audio contexts, addressing the severe token imbalance between the two modalities. OmniMem further preserves informative and non-redundant KV states through perturbation-aware memory selection, enabling compact memory without sacrificing long-range understanding. To strengthen compression under realistic deployment constraints, we also explore budget-aware fine-tuning, which encourages the model to consolidate useful information into retained memory. Experiments on VideoMME Long, LVBench, and LVOmniBench with video-SALMONN 2+ and Qwen-2.5-Omni show that OmniMem consistently improves over strong training-free compression baselines by 2-4% absolute accuracy under the same memory budgets, with an additional 1-2% gain after fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。