为SAM训练设计高效数据贡献评估方法,提升模型可解释性与鲁棒性。
Attributing Data for Sharpness-Aware Minimization
- 基于海塞矩阵与梯度轨迹,提出两种适配SAM双层结构的数据估值方法
- 可精准识别错误标注样本,支持模型编辑与参数调优,实验验证有效
- 适用于数据清洗、模型调试场景,尤其适合关注训练数据质量的研究者
Sharpness-aware Minimization (SAM) 通过关联损失曲面几何与泛化能力,提升了大规模模型训练的性能。然而,误标噪声数据与隐私问题日益突出。数据归属分析能够量化单个训练样本的贡献,提供可行解决方案。但直接将传统影响函数(IF)应用于SAM不可行,因其采用双层优化结构:内层寻找使损失最大化的模型扰动,外层最小化该最大损失,导致计算结构加倍,也使数据对参数的影响建模复杂化。本文基于影响函数,提出两种创新的数据估值方法:基于海塞矩阵的IF与基于梯度轨迹的IF。前者仅依赖训练后模型权重,提供闭式解析估计;后者利用训练过程中的梯度轨迹信息,实现更准确高效的评估。大量实验证明其在数据评估与参数调整中的有效性,可用于识别误标数据、模型编辑与提升可解释性。
原文摘要 · Abstract (English)
Sharpness-aware Minimization (SAM) improves generalization in large-scale model training by linking loss landscape geometry to generalization. However, challenges such as mislabeled noisy data and privacy concerns have emerged as significant issues. Data attribution, which identifies the contributions of specific training samples, offers a promising solution. However, directly rendering existing data influence evaluation tools such as influence functions (IF) to SAM will be inapplicable or inaccurate as SAM utilizes an inner loop to find model perturbations that maximize loss, which the outer loop then minimizes, resulting in a doubled computational structure. Additionally, this bilevel structure complicates the modeling of data influence on the parameters. In this paper, based on the IF, we develop two innovative data valuation methods for SAM, each offering unique benefits in different scenarios: the Hessian-based IF and the Gradient Trajectory-based IF. The first one provides a comprehensive estimation of data influence using a closed-form measure that relies only on the trained model weights. In contrast, the other IF for SAM utilizes gradient trajectory information during training for more accurate and efficient data assessment. Extensive experiments demonstrate their effectiveness in data evaluation and parameter tuning, with applications in identifying mislabeled data, model editing, and enhancing interpretability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。