arXiv:2512.10324cs.CV2025-12被引 3

解决音视频大模型冗余数据压缩中的位置混淆问题,实现高效低耗的联合降维。

EchoingPixels: Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMs

  • 通过跨模态语义筛分动态分配音视频压缩预算,提升信息保留率。
  • 在仅用原数据5%-20%令牌的情况下,性能接近全量模型。
  • 提出同步旋转位置编码,避免稀疏采样导致的时间相位错乱。

音视频大语言模型(AV-LLMs)面临处理海量冗余音视频令牌带来的高昂计算成本。现有单模态压缩方法无法捕捉联合音视频信号中异构且相互影响的信息密度。我们揭示了稀疏令牌缩减中的根本性理论瓶颈:位置混叠。研究表明,在标准位置编码序列上进行激进的稀疏采样,违反了相对于有效令牌间隔的奈奎斯特极限,引发相位包裹碰撞,破坏时间单调性。为此,我们提出EchoingPixels框架,实现抗混叠的联合令牌缩减。其跨模态语义筛分模块在协同音视频流上进行选择性提取,基于联合模态显著性动态分配压缩预算,而非固定每模态比例。为解决位置混叠问题,我们推导出Sync-RoPE,一种针对旋转位置编码的频谱低通滤波器,可根据稀疏采样率自适应调整编码带宽,保持缩减后序列的时间单调关系。实验表明,EchoingPixels仅使用原令牌的5%-20%即可达到与全模型相当的性能,验证了理论驱动的稀疏学习作为高效AV-LLMs的稳健解决方案。代码已开源于https://github.com/CharlesGong12/EchoingPixels。

原文摘要 · Abstract (English)

Audio-Visual Large Language Models (AV-LLMs) face prohibitive computational costs of processing massive, redundant audio-visual tokens. Existing unimodal compression techniques fail to capture the heterogeneous and mutually influential information density of joint audio-visual signals. Furthermore, we identify a fundamental and overlooked theoretical bottleneck in sparse token reduction: positional aliasing. We demonstrate that aggressive sparse sampling on standard position-encoded sequences violates the Nyquist limit relative to the effective token interval, causing phase-wrapping collisions that corrupt temporal monotonicity. To address this, we introduce EchoingPixels, a framework for aliasing-resistant joint token reduction. Our Cross-Modal Semantic Sieve performs extractive selection on the synergistic audio-visual stream, dynamically allocating budgets based on joint-modality saliency rather than fixed per-modality ratios. To resolve positional aliasing, we derive Sync-RoPE, a spectral low-pass filter for Rotary Positional Embeddings that adapts encoding bandwidth to the sparse sampling rate, preserving monotonic temporal relationships in the reduced stream. Experiments show that EchoingPixels achieves performance comparable to full models using only 5-20% of original tokens, validating theoretically grounded sparse learning as a robust solution for efficient AV-LLMs. Code is available at https://github.com/CharlesGong12/EchoingPixels.

音视频融合令牌压缩位置编码

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。