LLM在棋类游戏中推理会抑制策略多样性，导致提前崩溃。
When Reasoning Narrows the Move: Diversity Collapse in LLM Game Play
- 用井字棋变体测试模型决策多样性，发现推理模式降低动作多样性。
- 标准微调提升准确率却引发过早多样性衰减，超出必要牺牲范围。
- 多最优动作训练可部分缓解问题，利于保持探索性行为。
监督微调（SFT）广泛用于适配大语言模型至下游任务，但其对序列决策中行为多样性的影响仍不明确。我们在基于井字棋变体的可控环境中研究该问题，其中最优动作可精确计算，多样性可直接测量。在状态级评估、对战环境和训练轨迹中，我们发现推理模式常抑制动作多样性，且未统一提升动作准确性。此外，标准SFT虽提高准确率，但往往引发过早的多样性崩溃，超过准确率-多样性权衡所需的最小程度。随后我们证明，动作增强（即每状态训练所有最优动作而非单一示范动作）可部分缓解此问题。结果表明，窄支持模仿是大语言模型决策中策略坍缩的来源，提示在SFT中保留动作支持对维持探索行为至关重要。
原文摘要 · Abstract (English)
Supervised fine-tuning (SFT) is widely used to adapt large language models to downstream tasks, but its effect on behavioral diversity in sequential decision-making remains under-explored. We study this question in a controlled suite of deterministic board games based on tic-tac-toe variants, where optimal actions are exactly computable and diversity can be measured directly. Across state-level evaluation, arena gameplay, and training trajectories, we find that reasoning-mode generation frequently suppresses action diversity without uniformly improving action accuracy. Furthermore, standard SFT improves accuracy but often induces premature diversity collapse, which exceeds what is minimally required by the accuracy-diversity tradeoff. We then show that action augmentation, which trains on all optimal actions per state rather than a single demonstrated action, would partially mitigates this effect. Our results identify narrow-support imitation as a source of policy collapse in LLM decision-making and suggest that preserving action support during SFT is important for maintaining exploratory behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。