通过筛选低质量提示词，显著提升大模型性能。
R.I.P.: Better Models by Survival of the Fittest Prompts
- 基于拒绝响应质量与奖励差距评估提示词质量。
- 使用RIP过滤后，模型在多个基准上性能提升超9%。
- 适合想优化训练数据的模型开发者和研究者。
训练数据质量是影响最终模型质量的关键因素。本文提出一种基于假设的方法：低质量提示词会导致响应质量差且方差大。通过测量被拒绝响应的质量以及选择与拒绝偏好对之间的奖励差距，实现对数据完整性的评估。该方法名为拒绝指令偏好（Rejecting Instruction Preferences, RIP），可用于过滤现有训练集中的提示词，或生成高质量合成数据集。相较于未过滤数据，RIP在多个基准测试中均带来显著性能提升。使用 Llama 3.1-8B-Instruct 时，AlpacaEval2 LC Win Rate 提升 9.4%，Arena-Hard 提升 8.7%，WildBench 提升 9.9%。使用 Llama 3.3-70B-Instruct 时，Arena-Hard 得分从 67.5 提升至 82.9，排名由第18位跃升至第6位。
原文摘要 · Abstract (English)
Training data quality is one of the most important drivers of final model quality. In this work, we introduce a method for evaluating data integrity based on the assumption that low-quality input prompts result in high variance and low quality responses. This is achieved by measuring the rejected response quality and the reward gap between the chosen and rejected preference pair. Our method, Rejecting Instruction Preferences (RIP) can be used to filter prompts from existing training sets, or to make high quality synthetic datasets, yielding large performance gains across various benchmarks compared to unfiltered data. Using Llama 3.1-8B-Instruct, RIP improves AlpacaEval2 LC Win Rate by 9.4%, Arena-Hard by 8.7%, and WildBench by 9.9%. Using Llama 3.3-70B-Instruct, RIP improves Arena-Hard from 67.5 to 82.9, which is from 18th place to 6th overall in the leaderboard.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。