用可微注意力增强推理，让大模型的思考过程更真实可信。
AtManRL: Towards Faithful Reasoning via Differentiable Attention Saliency
- 通过可微注意力掩码识别推理中关键的词元
- 在GSM8K和MMLU上提升推理透明度与正确率
- 适合关注模型可解释性的研究人员
大语言模型越来越多地依赖思维链（CoT）推理来解决复杂任务，但确保推理过程不仅服务于最终答案，且真实反映模型内部决策机制仍具挑战。我们提出AtManRL，通过可微注意力操控结合强化学习，学习更忠实的推理过程。训练一个加性注意力掩码以识别影响正确答案的关键推理词元，并据此构建显著性奖励信号，促使模型生成真正影响预测的推理轨迹。该方法在GRPO框架中与基于结果的奖励联合优化，兼顾准确性与可解释性。在Llama-3.2-3B-Instruct上对GSM8K和MMLU的实验表明，该方法能有效识别关键推理词元，并训练出更具透明性的推理模型。
原文摘要 · Abstract (English)
Large language models (LLMs) increasingly rely on chain-of-thought (CoT) reasoning to solve complex tasks. Yet ensuring that the reasoning trace both contributes to and faithfully reflects the processes underlying the model's final answer, rather than merely accompanying it, remains challenging. We introduce AtManRL, a method that leverages differentiable attention manipulation to learn more faithful reasoning through reinforcement learning. By training an additive attention mask that identifies tokens in the CoT crucial for producing correct answers, we derive a saliency reward signal that encourages the model to generate reasoning traces that genuinely influence its final predictions. We integrate this saliency reward with outcome-based rewards within the GRPO framework to jointly optimize for correctness and interpretability. Experiments on GSM8K and MMLU with Llama-3.2-3B-Instruct demonstrate that our approach can identify influential reasoning tokens and enable training more transparent reasoning models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。