用大模型自动生成有挑战性的评测题，无需人工参与。
How to Get Your LLM to Generate Challenging Problems for Evaluation
- 自底向上构建难题，由简单组件逐步合成复杂问题。
- 生成的评测题使顶尖大模型准确率仅40%-60%，验证难度可信。
- 适用于评测文档问答、代码补全和数学推理任务，开源可用。
大语言模型的快速演进要求更严格全面的评估方法。传统人工标注因复杂性和成本已难以为继。本文提出CHASE框架，完全无需人工参与，利用大模型自动生成具有挑战性的评测题。针对特定任务，该方法从简单组件自底向上构建难题，并将生成过程拆分为可独立验证的子任务，确保质量与正确性。我们基于此框架在三个不同领域构建了评估基准：(1) 文档问答，(2) 代码库级代码补全，(3) 数学推理。当前顶尖大模型在这些合成基准上的表现准确率仅为40%-60%，充分证明了该框架生成高难度题的有效性。相关基准与代码已公开发布。
原文摘要 · Abstract (English)
The pace of evolution of Large Language Models (LLMs) necessitates new approaches for rigorous and comprehensive evaluation. Traditional human annotation is increasingly impracticable due to the complexities and costs involved in generating high-quality, challenging problems. In this work, we introduce CHASE, a unified framework to synthetically generate challenging problems using LLMs without human involvement. For a given task, our approach builds a hard problem in a bottom-up manner from simpler components. Moreover, our framework decomposes the generation process into independently verifiable sub-tasks, thereby ensuring a high level of quality and correctness. We implement CHASE to create evaluation benchmarks across three diverse domains: (1) document-based question answering, (2) repository-level code completion, and (3) math reasoning. The performance of state-of-the-art LLMs on these synthetic benchmarks lies in the range of 40-60% accuracy, thereby demonstrating the effectiveness of our framework at generating challenging problems. We publicly release our benchmarks and code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。