用跳棋游戏测试大模型决策能力，发现其策略常偏离最优解。
LUDOBENCH: Evaluating LLM Behavioural Decision-Making Through Spot-Based Board Game Scenarios in Ludo
- 设计480个棋局场景，分离出12类战略决策行为。
- 所有模型仅40%-46%与博弈论最优策略一致，分两类行为模式。
- 提示词变化会显著影响模型决策，暴露其敏感性弱点。
我们提出LudoBench，一个用于评估大语言模型在跳棋游戏中战略推理能力的基准。跳棋具有随机性、多智能体特性，包含掷骰子、吃子、安全格移动和归家路径推进，带来真实规划复杂性。LudoBench包含480个手工设计的棋局场景，覆盖12种行为上不同的决策类别，每类聚焦特定战略选择。我们还开发了一个支持随机、启发式、博弈论及大模型智能体的完整四人跳棋模拟器。博弈论代理采用期望极小极大搜索并结合深度有限前瞻，提供超越贪心启发法的严谨策略上限。评估六个跨四类模型家族的模型，发现所有模型仅在40%-46%的情况下与博弈论基线一致。模型呈现明显的行为类型：一类专注完成棋子但忽视发展，另一类专注发展却从不完成。每类仅捕捉博弈论策略的一半。模型在相同棋局下，因历史条件引发的怨恨提示而产生可测量的行为偏移，揭示提示敏感性是关键脆弱点。LudoBench为在不确定性下评估大模型战略推理提供了轻量且可解释的框架。所有代码、棋局数据集（480项）和模型输出均开源。
原文摘要 · Abstract (English)
We introduce LudoBench, a benchmark for evaluating LLM strategic reasoning in Ludo, a stochastic multi-agent board game whose dice mechanics, piece capture, safe-square navigation, and home-path progression introduce meaningful planning complexity. LudoBench comprises 480 handcrafted spot scenarios across 12 behaviorally distinct decision categories, each isolating a specific strategic choice. We additionally contribute a fully functional 4-player Ludo simulator supporting Random, Heuristic, Game-Theory, and LLM agents. The game-theory agent uses Expectiminimax search with depth-limited lookahead to provide a principled strategic ceiling beyond greedy heuristics. Evaluating six models spanning four model families, we find that all models agree with the game-theory baseline only 40-46% of the time. Models split into distinct behavioral archetypes: finishers that complete pieces but neglect development, and builders that develop but never finish. Each archetype captures only half of the game theory strategy. Models also display measurable behavioral shifts under history-conditioned grudge framing on identical board states, revealing prompt-sensitivity as a key vulnerability. LudoBench provides a lightweight and interpretable framework for benchmarking LLM strategic reasoning under uncertainty. All code, the spot dataset (480 entries) and model outputs are available at https://anonymous.4open.science/r/LudoBench-5CBF/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。