用合成数据提升大模型评分的效率与多样性
Generate, Evaluate, Iterate: Synthetic Data for Human-in-the-Loop Refinement of LLM Judges
- 通过可配置参数自动生成多样化测试用例
- 用户研究显示83%偏好该工具，效果媲美人工数据
- 适合需要快速迭代评分标准的研究者
LLM作为评分员的范式虽灵活，但受限于多样且具代表性的数据稀缺。本文提出一种集成合成数据生成的工具，支持用户配置领域、角色、长度和目标结果（包括边缘案例），实现定制化挑战性测试用例生成，并提供AI辅助的内联编辑功能。为增强透明度，系统展示每条生成背后的提示与解释。在24名用户的实验中，83%参与者更倾向使用该工具，因其能无额外负担快速生成多样化合成数据。生成数据在优化评估标准和对齐人类偏好方面效果等同于手工数据，证明合成数据是高效与可扩展场景下的可行替代方案。
原文摘要 · Abstract (English)
The LLM-as-a-judge paradigm enables flexible, user-defined evaluation, but its effectiveness is often limited by the scarcity of diverse, representative data for refining criteria. We present a tool that integrates synthetic data generation into the LLM-as-a-judge workflow, empowering users to create tailored and challenging test cases with configurable domains, personas, lengths, and desired outcomes, including borderline cases. The tool also supports AI-assisted inline editing of existing test cases. To enhance transparency and interpretability, it reveals the prompts and explanations behind each generation. In a user study (N=24), 83% of participants preferred the tool over manually creating or selecting test cases, as it allowed them to rapidly generate diverse synthetic data without additional workload. The generated synthetic data proved as effective as hand-crafted data for both refining evaluation criteria and aligning with human preferences. These findings highlight synthetic data as a promising alternative, particularly in contexts where efficiency and scalability are critical.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。