提出任务对齐代理，加速多任务视觉模型合并的超参选择。
Task Alignment: A Simple Proxy for Practical Model Merging Across Diverse Vision Tasks
- 用任务对齐度作为代理指标，替代耗时的下游训练。
- 在多个视觉任务上实现高效合并，性能接近原生模型。
- 适合需要快速整合多任务模型的工业级应用。
高效合并基于同一预训练基础模型、但微调于不同任务的多个模型，在实践中具有重要意义。尽管已有大量研究，但计算机视觉中的模型合并评估大多局限于使用CLIP进行图像分类的任务，其中不同分类数据集定义不同任务。本文旨在使模型合并更具实用性，并展示其在超越此类特定场景的挑战性任务中的价值。在多数视觉场景中，不同任务依赖于可训练且通常异构的解码器。与以往解码器冻结的研究不同，合并后需重新训练解码器，使得基于下游性能选择超参数变得不切实际。为此，我们提出任务对齐代理，证明其可将超参数选择速度提升数个数量级，同时保持性能。借助该代理，我们将模型合并的应用范围扩展至非CLIP的多任务视觉模型。
原文摘要 · Abstract (English)
Efficiently merging several models fine-tuned for different tasks, but stemming from the same pretrained base model, is of great practical interest. Despite extensive prior work, most evaluations of model merging in computer vision are restricted to image classification using CLIP, where different classification datasets define different tasks. In this work, our goal is to make model merging more practical and show its relevance on challenging scenarios beyond this specific setting. In most vision scenarios, different tasks rely on trainable and usually heterogeneous decoders. Differently from previous studies with frozen decoders, where merged models can be evaluated right away, the non-trivial cost of decoder training renders hyperparameter selection based on downstream performance impractical. To address this, we introduce the task alignment proxy, and show how it can be used to speed up hyperparameter selection by orders of magnitude while retaining performance. Equipped with the task alignment proxy, we extend the applicability of model merging to multi-task vision models beyond CLIP-based classification. Project page: https://europe.naverlabs.com/task-alignment
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。