探究思维链训练如何提升大模型智能体，发现其主要增强的是直接推理能力。
Where Do CoT Training Gains Land in LLM based Agents?
- 对比有无思维链的推理路径，发现训练更优的是直接从提示预测动作
- 思维链训练使提示动作质量显著提升，但未扩大思维链的相对优势
- 选择性屏蔽部分动作监督可提升模型在新场景下的泛化能力
思维链（CoT）推理广泛用于语言模型智能体，但已有研究指出，显式思维链未必忠实反映真实推理过程，可能只是事后解释。因此我们探讨：思维链训练究竟提升了什么？是让模型通过生成推理改变行为，还是更擅长直接从提示预测动作？通过对比仅基于提示的动作预测（prompt actions）与结合思维链的动作预测（CoT actions），我们发现随着模型迭代，提示动作的质量显著提升。在与环境交互过程中，思维链动作相对于提示动作的优势基本保持不变，表明思维链训练并未扩大其相对优势，反而有助于提升提示动作的性能。进一步发现，后期模型更少根据思维链修改动作，表现出对提示更强的依赖。基于此现象，我们对部分训练样本的选择性屏蔽动作令牌监督，该干预有效提升了模型在跨域任务中的泛化表现。
原文摘要 · Abstract (English)
Chain-of-thought (CoT) reasoning is widely used in language-model agents, but prior work has shown that verbalized CoT is not always faithful and may instead reflect post-hoc reasoning, which means the model already knows the answer before reasoning. We therefore ask what CoT training is actually improving: is the model getting better at changing its action through generated reasoning, or is it getting better at predicting the action directly from the prompt? We study this question by comparing \emph{prompt actions} (predicting action without CoT) with CoT actions (predicting action with CoT). Across checkpoints, prompt-action quality improves substantially. While interacting with the environment, the relative advantage of CoT actions over prompt actions remains similar, showing that CoT training does not widen the advantage of CoT reasoning, and it helps to improve the quality of prompt actions. We further find that later checkpoints are less likely to revise the action in response to CoT, suggesting greater reliance on the prompt. Motivated by these patterns, we selectively mask action-token supervision on a fraction of training examples. This intervention improves out-of-domain generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。