arXiv:2605.21422cs.LG2026-05

根据模型偏好权重选择数据,让微调更高效精准。

PRISM: Preference-Aware Influence Function Based Data Selection Method for Efficient Fine-Tuning

论文配图:PRISM: Preference-Aware Influence Function Based Data Selection Method for Efficient Fine-Tuning
图 1 · 摘自论文原文
  • 用模型偏好给目标样本加权,构建更优优化方向。
  • 在多个模型上实现更好性能,提升微调效率与对齐效果。
  • 适合追求高效、安全对齐微调的研究者和工程师。

随着大语言模型持续扩大规模,提升训练效率的关键在于有效利用数据。数据选择通过将有限的训练预算分配给高价值样本,以最优方式促进模型达到目标行为。现有方法通常通过一组目标样本来定义目标行为,并基于其对这些样本的影响估计来评分候选数据。然而,这些方法将所有目标样本视为同等重要,忽略了个体样本对模型优化的相关性差异:与模型固有行为一致的样本提供更强监督信号,而偏离的样本仅产生弱且无效的局部指导。本文提出 PRISM,一种基于偏好感知影响函数的数据选择方法。它利用模型偏好为目标样本赋予权重,构建偏好感知的目标方向,并根据候选数据对该方向的影响进行评估,优先将数据预算分配给能有效引导模型匹配预期目标行为的样本。理论分析表明,加权偏好构造比均匀聚合策略生成更优的一阶梯度方向,有助于增强目标偏好。大量实验涵盖多种模型架构与参数规模,验证了 PRISM 在高效微调及安全对齐监督微调修正中表现更优。结果表明,准确刻画目标行为是实现低成本数据选择的核心。

原文摘要 · Abstract (English)

As LLMs continue to scale up, improving training efficiency heavily relies on effective data utilization. Data selection mitigates this issue by allocating the limited training budget to high-value examples that optimally facilitate the model's target behavior. Most existing approaches define target behavior via a set of target examples and score candidate training data based on their estimated influence on these samples. However, such methods uniformly treat all target examples as equally important, ignoring the varying relevance of individual examples to model optimization. Specifically, target examples that align closely with the model's inherent behavior deliver stronger supervisory signals, whereas discrepant examples yield only weak and ineffective local guidance. We propose PRISM, a Preference-aware Influence function based Data Selection Method. It leverages model preference to assign weights to target examples and builds a preference-aware target direction. PRISM evaluates candidate training samples according to their influence on this direction, and prioritizes data budget allocation to samples that effectively drive the model to match expected target behavior. Theoretical analysis verifies that weighted preference construction generates a superior first-order gradient direction for boosting target preference, compared with uniform aggregation strategies. Extensive experiments covering diverse model architectures and parameter scales demonstrate that PRISM achieves better performance in efficient fine-tuning and safety-aligned supervised fine-tuning rectification. The results validate that accurate characterization of target behavior serves as the core of cost-effective data selection.

数据选择微调效率偏好学习模型对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。