arXiv:2508.11944cs.AIcs.CL2025-08被引 3

用认知层次模型评估大模型的策略推理能力,更可靠。

CHBench: A Cognitive Hierarchy Benchmark for Evaluating Strategic Reasoning Capability of LLMs

  • 基于行为经济学的认知层次模型设计评估框架
  • 六种大模型在十五个博弈中表现出稳定推理层级
  • 对话机制削弱策略推理,记忆机制则提升表现

博弈能力可反映大语言模型(LLMs)的战略推理水平。现有研究多依赖效用性能指标,但受对手行为和游戏结构差异影响,结果不够稳健。为此,我们提出新型评估框架——认知层次基准(CHBench),灵感来自行为经济学中的认知层次模型。假设智能体具有有限理性,不同智能体处于不同的推理深度。通过三阶段系统框架,利用六种先进大模型在十五个精心选择的常规模式博弈中的行为数据,评估其战略推理能力。实验表明,大模型在多种对手下表现出一致的推理层级,验证了该框架的稳健性与泛化能力。我们还分析了两种关键机制(对话机制与记忆机制)对推理性能的影响:结果显示,对话机制显著削弱战略推理,而记忆机制则有效增强。这些发现使CHBench成为评估大模型能力的有力工具,具备广阔的研究与应用前景。

原文摘要 · Abstract (English)

Game-playing ability serves as an indicator for evaluating the strategic reasoning capability of large language models (LLMs). While most existing studies rely on utility performance metrics, which are not robust enough due to variations in opponent behavior and game structure. To address this limitation, we propose \textbf{Cognitive Hierarchy Benchmark (CHBench)}, a novel evaluation framework inspired by the cognitive hierarchy models from behavioral economics. We hypothesize that agents have bounded rationality -- different agents behave at varying reasoning depths/levels. We evaluate LLMs' strategic reasoning through a three-phase systematic framework, utilizing behavioral data from six state-of-the-art LLMs across fifteen carefully selected normal-form games. Experiments show that LLMs exhibit consistent strategic reasoning levels across diverse opponents, confirming the framework's robustness and generalization capability. We also analyze the effects of two key mechanisms (Chat Mechanism and Memory Mechanism) on strategic reasoning performance. Results indicate that the Chat Mechanism significantly degrades strategic reasoning, whereas the Memory Mechanism enhances it. These insights position CHBench as a promising tool for evaluating LLM capabilities, with significant potential for future research and practical applications.

策略推理大模型评估博弈论

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。