不访问模型权重也能高效微调，适合保护隐私的场景。
Task-Specific Adaptation with Restricted Model Access
- 仅通过输入输出层插入轻量模块，实现灰箱微调。
- 在多任务基准上表现接近全权访问微调效果。
- 适合边缘设备与需保护模型版权的场景。
基础模型的兴起显著提升了各类下游任务的性能，微调通常能进一步提升效果。然而，现有微调方法大多需要访问模型权重和层，导致管理多个模型副本或推理管线困难，边缘设备优化效率低，且存在知识产权、隐私及不安全模型变体暴露等问题。本文探索‘灰箱’微调方法，模型架构与权重保持隐藏，仅允许梯度传播。提出一种新颖但简单有效的框架，在模型输入与输出端引入两个轻量可学习模块以适应新任务。此外，还设计了一种限制更少的变体，提供更多接入点，在性能与模型暴露间取得平衡。我们在多个骨干网络上评估了该方法，涵盖文本-图像对齐、文本-视频对齐与草图-图像对齐等任务。结果表明，尽管模型访问受限，灰箱方法仍具备与全权访问微调相当的竞争力。
原文摘要 · Abstract (English)
The emergence of foundational models has greatly improved performance across various downstream tasks, with fine-tuning often yielding even better results. However, existing fine-tuning approaches typically require access to model weights and layers, leading to challenges such as managing multiple model copies or inference pipelines, inefficiencies in edge device optimization, and concerns over proprietary rights, privacy, and exposure to unsafe model variants. In this paper, we address these challenges by exploring "Gray-box" fine-tuning approaches, where the model's architecture and weights remain hidden, allowing only gradient propagation. We introduce a novel yet simple and effective framework that adapts to new tasks using two lightweight learnable modules at the model's input and output. Additionally, we present a less restrictive variant that offers more entry points into the model, balancing performance with model exposure. We evaluate our approaches across several backbones on benchmarks such as text-image alignment, text-video alignment, and sketch-image alignment. Results show that our Gray-box approaches are competitive with full-access fine-tuning methods, despite having limited access to the model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。