发现大模型代理的决策忠实度问题主要出在推理到结论环节，而非执行阶段。
Doing What They Say, Not What They Reason: Locating the Faithfulness Gap in LLM Agents
- 拆解忠实度为推理到结论、结论到执行两步，分别测量
- 结论到执行几乎可靠（错误率0.7%~1.4%），但推理到结论错误率达22%-26%
- 模型会倾向保守决策，强制机械执行规则可减少一半错误
大模型代理是否按其自述推理做出决策？这一过程忠实性问题在缺乏正确行为参考时难以衡量。本文在德州扑克模拟器中构建可控环境，以验证动作作为基准，将忠实度差距分解为两步：推理到结论（所陈述决策是否由自身推理得出）和结论到行动（是否执行所述结论）。结果显示，结论到行动高度可靠（Claude Haiku 4.5错误率0.7%，DeepSeek-Reasoner为1.4%），而自由文本提取结论时错误率达22%-26%。推理到结论是主要失真来源，错误类型分布不均：坏输入、边界情况与规则误用各占约三分之一。规则误用在Haiku中占解释性错误的三分之一，在DeepSeek中仅占8%。唯一稳定信号是方向性偏差：当模型误用自身规则时，99.5%倾向风险规避。通过指令要求机械执行规则，可使误用率从13.9%降至6.8%，合规性提升8个百分点。因此，评估应聚焦可机器验证的结论，并检测方向性偏差，而非假设单一上游故障。
原文摘要 · Abstract (English)
Do LLM agents act on the reasoning they state? This question of process fidelity is central to LLM-based social simulation, yet hard to measure where no reference for correct behavior exists. We study it in a controlled setting: a Texas Poker simulator with a verifiable reference action for every decision by splitting the faithfulness gap into two steps: reasoning-to-conclusion (does the stated decision follow from the agent's own reasoning?) and conclusion-to-action (does the agent execute what it states?). The two steps behave very differently. Conclusion-to-action is reliable: inconsistency is 0.7% for Claude Haiku 4.5 and 1.4% for DeepSeek-Reasoner once the conclusion is read from an explicit tag, whereas free-text conclusion extraction reports 22-26%. Reasoning-to-conclusion is where fidelity frays, but not through a single dominant failure. In a step-level diagnostic the agent's errors split roughly evenly between bad inputs, borderline cases, and rule misapplication deriving a conclusion that contradicts the agent's own restated rule from inputs it estimated correctly. This composition is model-dependent: rule misapplication accounts for a third of Haiku's interpretable errors but only 8% of DeepSeek's. The one robust signal is directional: when an agent does misapply its own stated rule, it almost always (99.5% for Haiku) errs in the risk-averse direction. The override is partly hedging behavior, not a capability limit: instructing the agent to apply the rule mechanically halves the misapplication rate (13.9% to 6.8% of decisions) and raises adherence by eight points. Process-fidelity evaluation should therefore elicit machine-checkable conclusions and probe for directional biases rather than assume a single upstream failure mode, lest it conflate measurement noise with model behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。