用频域多尺度适配器，让模型更快更准地理解文本引导的图像数据。
Text-Guided Multi-Scale Frequency Representation Adaptation
- 在频域进行多尺度信号适配，融合文本信息提升表达能力。
- 仅用一个训练轮次就实现快速收敛，性能显著提升。
- 适合需要高效微调的多模态模型应用，如图文匹配与视觉理解。
参数高效微调方法通过引入少量可训练参数，使预训练模型能快速适应新数据分布。然而现有方法存在明显局限：多数在信号空间操作，导致信息冗余；且使用固定提示或适配层，未能充分捕捉信号的多尺度特性。为此，我们提出多尺度频域适配器（FreqAdapter），将文本信息融入频域中的多尺度信号微调过程，并设计多尺度适配策略以优化不同频段的感知范围，进一步增强模型表征能力。在包括CLIP和LLaVA在内的多个多模态模型上的大量实验表明，FreqAdapter在保持极低计算开销的同时，实现显著性能提升与快速收敛（仅需一个训练轮次）。代码已开源：https://github.com/Kelvin-ywc/FreqAdapter。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning methods introduce a small number of training parameters, enabling pre-trained models to adapt rapidly to new data distributions. While these methods have shown promising results, they exhibit notable limitations. First, most existing methods operate in the signal space domain, which results in substantial information redundancy. Second, most existing methods utilize fixed prompts or adaptation layers, failing to fully account for the multi-scale characteristics of signals. To address these challenges, we propose the Multi-Scale Frequency Adapter (FreqAdapter), which integrates textual information and performs multi-scale fine-tuning of signals in the frequency domain. Additionally, we introduce a multi-scale adaptation strategy to optimize receptive fields across different frequency ranges, further enhancing the model's representational capacity. Extensive experiments on multimodal models, including CLIP and LLaVA, demonstrate that FreqAdapter significantly improves both performance and efficiency. FreqAdapter improves performance with minimal cost and fast convergence within one epoch. Code is available at https://github.com/Kelvin-ywc/FreqAdapter.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。