用模型反馈优化合成数据评分标准,提升生成质量。
Optimsyn: Influence-Guided Rubrics Optimization for Synthetic Data Generation
- 基于模型梯度计算样本影响度,动态调整评分规则。
- 跨领域实验显示性能稳定提升,无需针对任务调参。
- 适合需要高质量合成数据的医疗、法律等专业领域。
大语言模型在下游任务中表现优异,主要得益于充足的监督微调(SFT)数据。然而,在人文、社科、医学、法律、金融等知识密集型领域,高质量SFT数据稀缺,因专家标注成本高、隐私限制严、标签一致性难保障。现有方法通常通过提示生成器并用手工设计的评分标准筛选输出,但评分标准依赖专家经验,跨领域迁移差,且优化过程依赖反复试错:撰写评分标准→生成数据→训练→人工检查→猜测修改。该流程缺乏量化反馈,无法准确评估评分标准对下游性能的影响。本文提出通过目标模型的训练效用评估合成数据,并利用此信号指导生成。受影响估计启发,采用基于梯度的优化感知估计器,量化每个合成样本对目标模型特定任务损失的贡献。分析表明,即使合成样本与真实样本在嵌入空间中相近,其学习影响也可能显著不同。据此,我们提出一种基于优化的框架,使用目标模型反馈自适应调整评分标准。提供轻量级引导文本,并采用专用评分模型生成任务条件化评分标准。以影响得分作为奖励,通过强化学习优化评分生成器。跨领域、多目标模型和生成器的实验表明,该方法持续提升性能,具备强泛化能力,且无需任务特异性调参。
原文摘要 · Abstract (English)
Large language models (LLMs) achieve strong downstream performance largely due to abundant supervised fine-tuning (SFT) data. However, high-quality SFT data in knowledge-intensive domains such as humanities, social sciences, medicine, law, and finance is scarce because expert curation is expensive, privacy constraints are strict, and label consistency is hard to ensure. Recent work uses synthetic data, typically by prompting a generator over domain documents and filtering outputs with handcrafted rubrics. Yet rubric design is expert-dependent, transfers poorly across domains, and is often optimized through a brittle heuristic loop of writing rubrics, synthesizing data, training, inspecting results, and manually guessing revisions. This process lacks reliable quantitative feedback about how a rubric affects downstream performance. We propose evaluating synthetic data by its training utility on the target model and using this signal to guide data generation. Inspired by influence estimation, we adopt an optimizer-aware estimator that uses gradient information to quantify each synthetic sample's contribution to a target model's objective on specific tasks. Our analysis shows that even when synthetic and real samples are close in embedding space, their influence on learning can differ substantially. Based on this insight, we propose an optimization-based framework that adapts rubrics using target-model feedback. We provide lightweight guiding text and use a rubric-specialized model to generate task-conditioned rubrics. Influence score is used as the reward to optimize the rubric generator with reinforcement learning. Experiments across domains, target models, and data generators show consistent improvements and strong generalization without task-specific tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。