用信息论自动生成高难度编程测试题，避免数据污染。
InfoSynth: Information-Guided Benchmark Synthesis for LLMs
- 基于信息熵和KL散度设计新颖性与多样性评估指标
- 97%成功率生成准确测试用例，难题比已有基准更难
- 可调节生成难度和多样性，适合评测大模型真实能力
大型语言模型在推理和代码生成方面取得显著进展，但高效构建新基准以评估这些能力仍具挑战。传统基准依赖人工，成本高且易污染训练数据。本文提出InfoSynth框架，基于信息论原则自动合成并评估推理基准。通过KL散度和熵设计无需昂贵模型评估的度量指标。基于此框架，开发端到端流水线，利用遗传算法和迭代代码反馈从种子数据集生成稳健的Python编程题。方法在97%情况下生成准确测试用例与解法，合成基准持续比先前工作更具挑战性。此外，算法支持对生成问题的新颖性/多样性与难度进行可控调节。InfoSynth提供可扩展、自验证的高质量编码基准构建方案。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated significant advancements in reasoning and code generation, but efficiently creating new benchmarks to evaluate these capabilities remains a challenge. Traditional benchmark creation relies on manual human effort, which is expensive and time-consuming. Furthermore, existing benchmarks often contaminate LLM training data, necessitating novel and diverse benchmarks to accurately assess their genuine capabilities. This work introduces InfoSynth, a novel framework for automatically generating and evaluating reasoning benchmarks guided by information-theoretic principles. We propose metrics based on KL-divergence and entropy to quantify benchmark novelty and diversity without relying on costly model evaluations. Building on this framework, we develop an end-to-end pipeline that synthesizes robust Python coding problems from seed datasets using genetic algorithms and iterative code feedback. Our method generates accurate test cases and solutions to new problems 97% of the time, and the synthesized benchmarks consistently exhibit higher difficulty compared to prior works. Moreover, our algorithm provides a method for controlling the novelty/diversity and difficulty of generated problems. InfoSynth offers a scalable, self-verifying pipeline for constructing high-quality, challenging coding benchmarks for LLMs. Project Page: https://ishirgarg.github.io/infosynth_web/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。