arXiv:2511.10979cs.CVcs.AI2025-11被引 1

提出无需训练的PAS机制,解决视频大模型时间编码不一致问题。

PAS: A Training-Free Stabilizer for Temporal Encoding in Video LLMs

  • 通过多头相位偏移后聚合,平滑时间核响应
  • 在多个基准上提升性能,计算开销几乎为零
  • 适合所有基于旋转位置编码的视频大模型

视频大模型存在时间不一致性:帧时间微小偏移会导致注意力分配翻转并抑制相关帧。我们发现该问题源于将旋转位置编码(RoPE)扩展至视频时引入的逆傅里叶时间核,其在帧尺度产生波动,使相邻帧被不同因子放大,扰乱本应由原始查询-键内积主导的注意力。本文提出相位聚合平滑(PAS),一种简单、无需训练的机制:在各注意力头施加微小反向相位偏移后聚合输出。PAS保持各头频谱幅度不变,聚合过程有效平滑时间核,降低对相位敏感性,且不改变位置编码结构。分析表明,RoPE旋转后的logit可近似为内容点积乘以时间核;平滑该核可实现注意力对小时间偏移的利普希茨稳定性;多相位平均抑制高频波动,同时在奈奎斯特采样下保持各头频谱特性。在多个视频理解基准上,使用相同标记预算的实验显示,PAS持续提升性能,计算开销可忽略。PAS为视频大模型提供即插即用的时间鲁棒编码升级。

原文摘要 · Abstract (English)

Video LLMs suffer from temporal inconsistency: small shifts in frame timing can flip attention and suppress relevant frames. We trace this instability to the common extension of Rotary Position Embeddings to video through multimodal RoPE. The induced inverse Fourier time kernel exhibits frame-scale ripples that multiply adjacent frames by different factors, which perturbs attention that should otherwise be governed by the raw query key inner product. We present Phase Aggregated Smoothing (PAS), a simple, training-free mechanism that applies small opposed phase offsets across heads and then aggregates their outputs. PAS preserves the per-head spectrum magnitude, while the aggregation effectively smooths the temporal kernel and reduces phase sensitivity without changing the positional encoding structure. Our analysis shows that the RoPE rotated logit can be approximated as a content dot product scaled by a time kernel; smoothing this kernel yields Lipschitz stability of attention to small temporal shifts; multi phase averaging attenuates high frequency ripples while preserving per-head spectra under Nyquist-valid sampling. Experiments on multiple video understanding benchmarks under matched token budgets show consistent improvements with negligible computational overhead. PAS provides a plug and play upgrade for robust temporal encoding in Video LLMs.

视频生成位置编码注意力机制稳定性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。