用微分变换实现高效多任务微调，参数少75%还更准
DiTASK: Multi-Task Fine-Tuning with Diffeomorphic Transformations
- 通过保持权重矩阵奇异向量，仅变换奇异值实现任务特化
- 在PASCAL MTL和NYUD上4个密集预测任务均达领先性能
- 适合需要少参数多任务部署的视觉模型优化场景
预训练视觉变压器已成为计算机视觉的强大工具。然而，如何高效地将其适配到多个任务仍面临挑战，因为需修改丰富的隐藏表征，又不能引发任务间干扰。现有参数高效方法如LoRA采用低秩更新，迫使各任务在受限子空间中竞争，最终导致性能下降。我们提出DiTASK——一种基于微分同胚变换的新型多任务微调方法，通过保留权重矩阵的奇异向量来维持预训练表征，同时利用神经微分同胚变换对奇异值进行任务特异性调整。该方法实现了共享与任务特定特征调制，且新增参数极少。理论分析表明，DiTASK在优化过程中可实现全秩更新，保持预训练特征的几何结构，开创了高效多任务学习的新范式。在PASCAL MTL和NYUD数据集上的实验表明，DiTASK在四个密集预测任务上均达到当前最优表现，所用参数比现有方法减少75%。代码已开源。
原文摘要 · Abstract (English)
Pre-trained Vision Transformers now serve as powerful tools for computer vision. Yet, efficiently adapting them for multiple tasks remains a challenge that arises from the need to modify the rich hidden representations encoded by the learned weight matrices, without inducing interference between tasks. Current parameter-efficient methods like LoRA, which apply low-rank updates, force tasks to compete within constrained subspaces, ultimately degrading performance. We introduce DiTASK a novel Diffeomorphic Multi-Task Fine-Tuning approach that maintains pre-trained representations by preserving weight matrix singular vectors, while enabling task-specific adaptations through neural diffeomorphic transformations of the singular values. By following this approach, DiTASK enables both shared and task-specific feature modulations with minimal added parameters. Our theoretical analysis shows that DITASK achieves full-rank updates during optimization, preserving the geometric structure of pre-trained features, and establishing a new paradigm for efficient multi-task learning (MTL). Our experiments on PASCAL MTL and NYUD show that DiTASK achieves state-of-the-art performance across four dense prediction tasks, using 75% fewer parameters than existing methods. Our code is available [here](https://github.com/ipsitmantri/DiTASK).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。