用稀疏自编码器让CLIP微调更鲁棒且可解释
Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models
- 基于预训练模型的稀疏自编码器识别语义特征，约束视觉表征变化
- 微调后在ImageNet及分布外测试上性能达或超越当前最优
- 无需文本引导，计算高效且能直接分析语义变化
大规模预训练视觉语言模型如CLIP在多种任务上展现出卓越的零样本性能。然而，微调这些模型以提升下游表现时，常导致对分布偏移的鲁棒性下降。现有方法虽试图缓解此权衡，但多依赖计算开销大的文本引导。本文提出新方法SAE-FT，仅作用于模型的视觉表征。该方法利用在预训练模型上训练的稀疏自编码器识别出有意义的语义特征，通过惩罚表征中新增或删除这些特征来正则化变化，防止灾难性遗忘，并使微调过程可解释，支持直接分析语义改变。SAE-FT机制透明、计算高效，在ImageNet及其分布偏移基准上达到或超过当前最优性能。代码已公开于：https://github.com/Fabian-Mor/sae-ft。
原文摘要 · Abstract (English)
Large-scale pre-trained vision-language models like CLIP demonstrate remarkable zero-shot performance across diverse tasks. However, fine-tuning these models to improve downstream performance often degrades robustness against distribution shifts. Recent approaches have attempted to mitigate this trade-off, but often rely on computationally expensive text-guidance. We propose a novel method for robust fine-tuning, SAE-FT, which operates only on the model's visual representations. SAE-FT regularizes changes to these representations by penalizing the addition and removal of semantically meaningful features identified by a Sparse Autoencoder trained on the pre-trained model. This constraint prevents catastrophic forgetting and makes the fine-tuning process interpretable, enabling direct analysis of semantic changes. SAE-FT is both mechanistically transparent and computationally efficient, matching or exceeding state-of-the-art performance on ImageNet and its associated distribution shift benchmarks. Code is publicly available at: https://github.com/Fabian-Mor/sae-ft.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。