无需人工标注,用轮换评估法选出最适合生成合成数据的大模型。
RoSE: Round-robin Synthetic Data Evaluation for Selecting LLM Generators without Human Test Sets
- 让小模型在某个大模型生成的数据上训练,再测试其在其他大模型数据上的表现,取平均得分。
- 在11种语言、3类任务中,比其他内在指标更准地选出最优生成器。
- 唯一与真实人类测试表现正相关的方法,适合低资源语言场景。
大语言模型(LLM)能高效生成合成数据,用于训练小型专用模型,尤其对缺乏人工标注数据的低资源语言至关重要。然而不同LLM生成数据的质量差异显著,而传统外部评估需昂贵的人工标注(常不可得),内部指标又与下游性能关联弱。本文提出轮换合成数据评估(RoSE),一种无需人类测试集的代理指标:用候选生成器输出训练一个小模型,再在其余所有生成器的合成数据上评估该小模型,最终得分取均值。在六种大模型、十一种语言、三种任务(情感、主题、意图)下,RoSE比其他内在启发式方法更频繁识别出最优生成器。其选择结果的下游性能仅比最优基准低0.76个百分点,且是唯一与人类测试数据表现呈正相关的指标。
原文摘要 · Abstract (English)
LLMs are powerful generators of synthetic data, which are used for training smaller, specific models. This is especially valuable for low-resource languages, where human-labelled data is scarce but LLMs can still produce high-quality text. However, LLMs differ in how useful their outputs are for training. Selecting the best LLM as a generator is challenging because extrinsic evaluation requires costly human annotations (which are often unavailable for low-resource languages), while intrinsic metrics correlate poorly with downstream performance. We introduce Round robin Synthetic data Evaluation (RoSE), a proxy metric for selecting the best LLM generator without human test sets. RoSE trains a small model on the outputs of a candidate generator (LLM) and then evaluates it on generated synthetic examples from all other candidate LLMs. The final RoSE score is the mean performance of this small model. Across six LLMs, eleven languages, and three tasks (sentiment, topic, intent), RoSE identifies the optimal generator more often than any other intrinsic heuristics. RoSE outperforms intrinsic heuristics and comes within 0.76 percentage points of the optimal generator baseline. This result is measured in terms of downstream performance, obtained by training a small model on the chosen generator's outputs (optimal vs. proxy metric selected) and evaluating it on human-labelled test data. Additionally, RoSE is the only metric to achieve a positive correlation with performance on human test data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。