提出定位并编辑视觉Transformer错误参数的方法,提升模型对罕见样本的纠错能力。
Learning Where to Edit Vision Transformers
- 通过元学习生成可泛化的二值掩码,定位需编辑的稀疏参数
- 在包含罕见图像的基准上实现优于基线的纠错效果
- 支持灵活权衡泛化性与局部性,适合实际部署场景
模型编辑旨在以数据高效的方式纠正大模型的预测错误,同时保证对邻近错误的泛化能力,并限制对无关样本的副作用。尽管在基于Transformer的大语言模型编辑方面取得进展,视觉Transformer(ViTs)在计算机视觉中的有效编辑策略仍不成熟。本文针对由子群体偏移引发的预测错误,提出“定位-编辑”框架:首先在CutMix增强数据上通过元学习训练超网络,生成可靠的二值掩码,定位稀疏且结构化的模型参数;随后采用变体梯度下降法仅微调这些参数完成编辑。为验证方法,构建了一个引入自然罕见图像和AI生成图像的子群体偏移基准,揭示预训练ViT在目标识别中的局限性。实验表明,该方法在新基准上表现更优,并支持在泛化性与局部性之间调节。代码已开源。
原文摘要 · Abstract (English)
Model editing aims to data-efficiently correct predictive errors of large pre-trained models while ensuring generalization to neighboring failures and locality to minimize unintended effects on unrelated examples. While significant progress has been made in editing Transformer-based large language models, effective strategies for editing vision Transformers (ViTs) in computer vision remain largely untapped. In this paper, we take initial steps towards correcting predictive errors of ViTs, particularly those arising from subpopulation shifts. Taking a locate-then-edit approach, we first address the where-to-edit challenge by meta-learning a hypernetwork on CutMix-augmented data generated for editing reliability. This trained hypernetwork produces generalizable binary masks that identify a sparse subset of structured model parameters, responsive to real-world failure samples. Afterward, we solve the how-to-edit problem by simply fine-tuning the identified parameters using a variant of gradient descent to achieve successful edits. To validate our method, we construct an editing benchmark that introduces subpopulation shifts towards natural underrepresented images and AI-generated images, thereby revealing the limitations of pre-trained ViTs for object recognition. Our approach not only achieves superior performance on the proposed benchmark but also allows for adjustable trade-offs between generalization and locality. Our code is available at https://github.com/hustyyq/Where-to-Edit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。