arXiv:2603.20403cs.CV2026-03

提出自动调优秩的高效多任务微调方法,提升模型精度与效率。

FAAR: Efficient Frequency-Aware Multi-Task Fine-Tuning via Automatic Rank Selection

  • 根据任务和层位置动态分配适配器秩,实现精准参数控制。
  • 在密集视觉任务上比传统方法减少9倍参数,性能更优。
  • 通过频谱分析注入跨任务关联信息,适合多任务场景优化。

在大规模数据集上预训练模型是快速实现下游任务强性能的有效方式。然而,当前先进模型规模不断增长,传统全量微调已难适用,尤其在多任务学习(MTL)中,成本随任务数量线性上升。为此,近期研究采用参数高效微调(PEFT)技术,如低秩适应,大幅减少可训练参数。但现有方法使用单一固定秩,难以适配不同任务或架构位置。此外,这些方法未能有效捕捉空间信息以反映任务间关系,影响多样任务预测效果。本文提出频率感知且自动调优秩的高效多任务微调方法(FAAR)。我们设计性能驱动的秩压缩机制(PDRS),为每个适配器位置和任务分配最优秩;同时基于图像频谱分析,引入任务-频谱金字塔解码器(TS-PD),将输入特异性上下文注入空间偏置学习,更好表达跨任务关联。在密集视觉任务基准测试中,相比其他PEFT方法,FAAR在准确率与效率上均表现更优:相比传统多任务微调,参数量减少最多达9倍,整体性能反而提升。代码已公开。

原文摘要 · Abstract (English)

Adapting models pre-trained on large-scale datasets is a proven way to reach strong performance quickly for down-stream tasks. However, the growth of state-of-the-art mod-els makes traditional full fine-tuning unsuitable and difficult, especially for multi-task learning (MTL) where cost scales with the number of tasks. As a result, recent studies investigate parameter-efficient fine-tuning (PEFT) using low-rank adaptation to significantly reduce the number of trainable parameters. However, these existing methods use a single, fixed rank, which may not be optimal for differ-ent tasks or positions in the MTL architecture. Moreover, these methods fail to learn spatial information that cap-tures inter-task relationships and helps to improve diverse task predictions. This paper introduces Frequency-Aware and Automatic Rank (FAAR) for efficient MTL fine-tuning. Our method introduces Performance-Driven Rank Shrink-ing (PDRS) to allocate the optimal rank per adapter location and per task. Moreover, by analyzing the image frequency spectrum, FAAR proposes a Task-Spectral Pyramidal Decoder (TS-PD) that injects input-specific context into spatial bias learning to better reflect cross-task relationships. Experiments performed on dense visual task benchmarks show the superiority of our method in terms of both accuracy and efficiency compared to other PEFT methods in MTL. FAAR reduces the number of parameters by up to 9 times compared to traditional MTL fine-tuning whilst improving overall performance. Our code is available.

多任务学习参数高效低秩微调视觉任务

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。