对比微调与低秩适配器在相同学习率下的表现，发现后者更保真迁移能力。
Matched-Learning-Rate Analysis of Attention Drift and Transfer Retention in Fine-Tuned CLIP
- 在相同学习率下比较全参数微调与LoRA，控制变量分析注意力漂移与迁移保留。
- 相同学习率下LoRA在CIFAR-100零样本上平均达45.13%准确率，远超全微调的11.28%。
- 注意力熵变化可作表征保留的描述性诊断，但无法解释迁移性能差异。
CLIP微调能提升域内准确率，但会损害域外迁移性能，而全微调（Full FT）与LoRA的对比常因学习率设定不同而混淆。本文通过在相同学习率下系统比较Full FT与LoRA，研究方法与优化规模如何共同影响CLIP中的注意力漂移与迁移保留。实验覆盖CLIP ViT-B/32在EuroSAT与Oxford-IIIT Pets两个数据集上的80次运行，包含四个共享学习率（$10^{-6}$, $5{ imes}10^{-6}$, $10^{-5}$, $5{ imes}10^{-5}$）及五个随机种子，评估注意力漂移指标、最佳验证准确率与适配器感知的CIFAR-100零样本准确率。结果显示：学习率显著调控结构变化——在EuroSAT上，全微调从$10^{-6}$时的轻微熵扩散演变为$5{ imes}10^{-5}$时的明显收缩，而LoRA在整个匹配范围内保持熵增；在相同学习率下，LoRA的零样本迁移性能显著优于全微调，分别在EuroSAT和Pets上达到45.13%与58.01%，而全微调仅为11.28%与8.54%。此外，低学习率下LoRA在域内任务存在欠拟合现象，仅靠方法均值会掩盖其竞争性出现的时机。支持性分析（展开、像素对像素、CKA）方向一致。总体表明，匹配学习率评估显著改变对两种方法的评价，注意力漂移更适合作为表征保留的描述性诊断工具而非因果解释。
原文摘要 · Abstract (English)
CLIP adaptation can improve in-domain accuracy while degrading out-of-domain transfer, but comparisons between Full Fine-Tuning (Full FT) and LoRA are often confounded by different learning-rate conventions. We study how adaptation method and optimization scale jointly shape attention drift and transfer retention in CLIP using a controlled matched-learning-rate comparison of Full FT and LoRA. The completed matrix contains 80 runs on CLIP ViT-B/32 across EuroSAT and Oxford-IIIT Pets, spanning four shared learning rates ($10^{-6}$, $5{\times}10^{-6}$, $10^{-5}$, $5{\times}10^{-5}$) and five seeds, and evaluates attention-drift metrics, best validation accuracy, and adapter-aware CIFAR-100 zero-shot accuracy. Learning rate strongly modulates structural change: on EuroSAT, Full FT moves from mild entropy broadening at $10^{-6}$ to marked contraction at $5{\times}10^{-5}$, whereas LoRA remains entropy-positive across the full matched grid. At matched learning rates, LoRA preserves substantially more zero-shot transfer than Full FT, averaging $45.13\%$ versus $11.28\%$ CIFAR-100 accuracy on EuroSAT and $58.01\%$ versus $8.54\%$ on Pets. Oxford-IIIT Pets also reveals a regime effect: low-learning-rate LoRA underfits in-domain, so method-only averages can obscure when LoRA becomes competitive. Supporting rollout, patch-to-patch, and CKA analyses are directionally consistent with the controlled matrix. Overall, matched-learning-rate evaluation materially changes the interpretation of Full FT versus LoRA, and attention drift is most useful as a descriptive diagnostic of representation preservation rather than a causal explanation of transfer behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。