通过频谱编码提升视频理解精度，捕捉中速运动细节。
Frame2Freq: Spectral Adapters for Fine-Grained Video Understanding
- 引入频域适配器，用FFT分析多时间尺度动态
- 在5个数据集上超越现有方法，4个超全微调模型
- 适合需要精细动作识别的视频分析任务
将图像预训练主干网络迁移到视频任务通常依赖单一时间尺度的时域适配器，但实验表明这类模块仅捕捉静态图像特征和快速闪烁变化，忽略中速运动。而细粒度时间分析（如开瓶与关瓶）恰恰需要多尺度动态建模。为此，我们提出Frame2Freq——一种频谱感知适配器，在预训练视觉基础模型（VFMs）的图像到视频迁移中进行频域编码，提升细粒度动作识别性能。Frame2Freq沿时间维度使用快速傅里叶变换（FFT），学习各频带特定嵌入，自适应突出最具判别性的频率范围。在五个细粒度活动识别数据集上，Frame2Freq优于以往参数高效微调（PEFT）方法，并在四个数据集上超越全模型微调。结果表明频域分析是建模图像到视频迁移中时间动态的强大工具。代码已开源：https://github.com/th-nesh/Frame2Freq。
原文摘要 · Abstract (English)
Adapting image-pretrained backbones to video typically relies on time-domain adapters tuned to a single temporal scale. Our experiments show that these modules pick up static image cues and very fast flicker changes, while overlooking medium-speed motion. Capturing dynamics across multiple time-scales is, however, crucial for fine-grained temporal analysis (i.e., opening vs. closing bottle). To address this, we introduce Frame2Freq -- a family of frequency-aware adapters that perform spectral encoding during image-to-video adaptation of pretrained Vision Foundation Models (VFMs), improving fine-grained action recognition. Frame2Freq uses Fast Fourier Transform (FFT) along time and learns frequency-band specific embeddings that adaptively highlight the most discriminative frequency ranges. Across five fine-grained activity recognition datasets, Frame2Freq outperforms prior PEFT methods and even surpasses fully fine-tuned models on four of them. These results provide encouraging evidence that frequency analysis methods are a powerful tool for modeling temporal dynamics in image-to-video transfer. Code is available at https://github.com/th-nesh/Frame2Freq.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。