让大模型的推理过程更可信：通过多答案评估提升隐式推理的奖励分配精度。
Latent Thought Credit: Multi-Answer Credit Assignment for Latent Reasoning
- 用多轮生成答案来估算每个中间推理步骤的奖励，避免单次答案噪声干扰
- 在数学与STEM多选题上，准确率优于现有方法，平均提升3.2个百分点
- 适合研究大模型内部推理机制或想提升推理可解释性的研究人员
隐式推理允许语言模型在连续潜在表示中进行中间推理，而非显式输出离散思维链。然而，仅基于最终答案的奖励来分配隐式推理的信用存在困难：单一答案混合了推理质量与答样本噪声。我们提出【隐式思维信用（Latent Thought Credit, LTC）】，一种分层信用分配框架。对每个提示，LTC采样多个潜在推理步骤，在每一步后固定上下文，并通过从该固定上下文生成的多个答案的奖励均值估计该步骤的期望奖励。LTC使用推理级优势优化隐式推理阶段，答案级优势优化回答阶段，并采用优势加权的推理匹配目标，帮助策略重现高信用的潜在推理。我们在类似GRPO的在线策略训练框架中实现LTC，评估其在数学推理和STEM多选任务上的表现。LTC在对比方法中取得最高平均准确率；消融实验与固定上下文诊断显示，多答案估计显著降低奖励估计误差，缓解推理层级信用的模糊或错误问题。
原文摘要 · Abstract (English)
Latent reasoning allows language models to carry out intermediate reasoning in continuous latent representations rather than fully externalizing it as discrete chains of thought. However, assigning credit to such latent thoughts from answer-only rewards is difficult: a single final answer mixes thought quality with answer-sampling noise. We propose \textbf{Latent Thought Credit (LTC)}, a hierarchical credit-assignment framework for latent reasoning. For each prompt, LTC samples multiple latent thoughts, fixes the context after each thought, and estimates thought-level expected reward by averaging rewards over multiple answers generated from that fixed context. LTC uses thought-level advantages to optimize the latent-thought phase, answer-level advantages to optimize the answer phase, and an advantage-weighted thought-matching objective that helps the policy reproduce high-credit latent thoughts. We instantiate LTC in a GRPO-style on-policy training framework and evaluate it across mathematical reasoning and STEM multiple-choice tasks. LTC achieves the best average accuracy among the compared methods, while ablations and fixed-context diagnostics show that multi-answer estimation reduces reward-estimation error and mitigates ambiguous or incorrect thought-level credit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。