用影响函数筛选偏好数据，删10%样本提1.5%准确率
Influence Functions for Preference Dataset Pruning
- 用共轭梯度近似影响函数识别有害训练样本
- 删除10%样本后验证集准确率提升1.5%
- 局部曲率对检测有害样本重要，对好样本不敏感
语言模型常通过强化学习微调以改变行为或激发新能力。用于此类目的的数据集，尤其是人类偏好数据集，通常含有噪声。由于微调后数据集规模较小，结合参数高效微调方法，可使用影响函数近似来检测并剔除对验证集性能有害的训练样本。本文将TL;DR数据集用于奖励模型训练，展示如何利用共轭梯度近似的影响函数进行数据过滤。实验表明，剔除10%的训练样本后，重训练准确率提升1.5%。此外，我们发现梯度相似性在识别有益样本方面优于影响函数，表明局部曲率对检测有害样本更关键，而对识别有益样本作用较小。
原文摘要 · Abstract (English)
Language models are commonly fine-tuned via reinforcement learning to alter their behavior or elicit new capabilities. Datasets used for these purposes, and particularly human preference datasets, are often noisy. The relatively small size post-training datasets, combined with parameter-efficient fine-tuning methods, enable the use of influence functions approximations to detect and prune training examples that are harmful to performance on a validation set. In this work, we adapt the TL;DR dataset for reward model training to demonstrate how conjugate-gradient approximated influence functions can be used to filter datasets. In our experiments, influence function filtering yields a small retraining accuracy uplift of 1.5% after removing 10% of training examples. We also show that gradient similarity outperforms influence functions for detecting helpful training examples. This suggests that local curvature is important for detecting harmful training examples, but less so for identifying helpful examples.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。