用策略游戏评估大模型的推理过程，看清其规划与决策质量。
Tracing LLM Reasoning Processes with Strategic Games: A Framework for Planning, Revision, and Resource-Constrained Decision Making
- 设计策略游戏环境，量化模型的规划、修正与资源使用行为。
- 发现纠错频率高未必效果好，过度修正反而降低成功率。
- 适合关注模型可靠性与推理逻辑的研究者和开发者。
大语言模型在复杂推理任务中的应用日益广泛，但现有评测多聚焦最终结果，忽视了规划、修正及资源受限下的决策等中间过程。本文提出以策略游戏为自然评估环境：规则封闭、状态清晰、资源有限且具备自动反馈。构建框架从规划、修正和资源约束决策三方面评估模型。引入超越胜率的指标，包括过度修正风险率、修正成功率、改进斜率和超预算比例。在12个主流模型上进行4320轮对抗测试，ChatGPT-o3-mini综合得分最高，胜率74.7%，修正成功率78.6%，改进斜率0.041；而Qwen-Plus虽过度修正风险率达81.6%，胜率仅25.6%，主要因资源消耗过大。观察到过度修正风险率与修正成功率呈负相关（皮尔逊r = -0.51，p = 0.093），表明频繁修改不等于更好结果。研究强调，评估模型不仅要看决策结果，更需关注其推理路径。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly used for tasks that require complex reasoning. Most benchmarks focus on final outcomes but overlook the intermediate reasoning steps - such as planning, revision, and decision making under resource constraints. We argue that measuring these internal processes is essential for understanding model behavior and improving reliability. We propose using strategic games as a natural evaluation environment: closed, rule-based systems with clear states, limited resources, and automatic feedback. We introduce a framework that evaluates LLMs along three core dimensions: planning, revision, and resource-constrained decision making. To operationalize this, we define metrics beyond win rate, including overcorrection risk rate, correction success rate, improvement slope, and over-budget ratio. In 4320 adversarial rounds across 12 leading models, ChatGPT-o3-mini achieves the top composite score, with a win rate of 74.7 percent, a correction success rate of 78.6 percent, and an improvement slope of 0.041. By contrast, Qwen-Plus, despite an overcorrection risk rate of 81.6 percent, wins only 25.6 percent of its matches - primarily due to excessive resource use. We also observe a negative correlation between overcorrection risk rate and correction success rate (Pearson r = -0.51, p = 0.093), suggesting that more frequent edits do not always improve outcomes. Our findings highlight the value of assessing not only what LLMs decide but how they arrive at those decisions
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。