arXiv:2504.05500cs.AIcs.LG2025-04被引 3

用蒙特卡洛树搜索动态评估大模型代码生成能力,更精准发现模型短板。

Prism: Dynamic and Flexible Benchmarking of LLMs Code Generation with Monte Carlo Tree Search

  • 将评测建模为马尔可夫决策过程,用树搜索自动挖掘挑战性测试场景。
  • 在5个顶级大模型上验证,发现模型规模和架构显著影响不同难度任务表现。
  • 适合研究者用于深度分析模型缺陷,也适用于评估工具开发者参考。

大型语言模型(LLMs)的快速演进已超越传统评估方法。静态基准无法全面反映模型能力,且很快过时;现有动态方法或过度依赖大模型自身评估,或受限于预设测试集。我们提出Prism,一个灵活、动态的基准评测框架,用于全面评估大模型性能。Prism基于三个核心组件:(1) 基于树的状态表示,将评测建模为马尔可夫决策过程;(2) 适配的蒙特卡洛树搜索算法,用于发现具有挑战性的测试场景;(3) 多智能体评估流水线,实现对多种能力的并行评估。为确保评估稳健性,Prism结合树探索模式的结构度量与多难度层级的表现指标,提供错误模式、测试覆盖度及解法路径的详细诊断。在五个前沿大模型上进行大量实验,分析了模型架构与规模如何影响不同难度下的代码生成表现。结果表明,Prism作为动态基准能随模型演进而演化,并深入揭示其局限性。

原文摘要 · Abstract (English)

The rapid advancement of Large Language Models (LLMs) has outpaced traditional evaluation methods. Static benchmarks fail to capture the depth and breadth of LLM capabilities and eventually become obsolete, while most dynamic approaches either rely too heavily on LLM-based evaluation or remain constrained by predefined test sets. We introduce Prism, a flexible, dynamic benchmarking framework designed for comprehensive LLM assessment. Prism builds on three key components: (1) a tree-based state representation that models evaluation as a Markov Decision Process, (2) a Monte Carlo Tree Search algorithm adapted to uncover challenging evaluation scenarios, and (3) a multi-agent evaluation pipeline that enables simultaneous assessment of diverse capabilities. To ensure robust evaluation, Prism integrates structural measurements of tree exploration patterns with performance metrics across difficulty levels, providing detailed diagnostics of error patterns, test coverage, and solution approaches. Through extensive experiments on five state-of-the-art LLMs, we analyze how model architecture and scale influence code generation performance across varying task difficulties. Our results demonstrate Prism's effectiveness as a dynamic benchmark that evolves with model advancements while offering deeper insights into their limitations.

代码生成动态评估蒙特卡洛树搜索大模型评测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。