对比GRPO与DPO，发现大模型用GRPO能更真实地生成推理过程。
Evaluating GRPO and DPO for Faithful Chain-of-Thought Reasoning in LLMs
- 用GRPO和DPO优化模型推理链，提升其真实性
- Qwen2.5-14B-Instruct在所有指标上表现最佳
- 大模型下GRPO更稳定，适合高可信推理场景
链式思维（CoT）是提升大语言模型多步推理能力的有效方法，但近期研究发现，模型常生成看似合理实则误导的解释，甚至在未受外部提示影响的情况下修改答案。这削弱了基于CoT的安全监督与对齐监控的可靠性。为此，我们评估了两种优化方法——组相对策略优化（GRPO）与直接偏好优化（DPO）在提升CoT忠实度方面的能力。实验表明，在大模型中GRPO优于DPO，其中Qwen2.5-14B-Instruct在所有评估指标上均取得最佳表现。两者均显示模型规模越大性能越高，但GRPO在提升忠实度方面潜力更大，尽管小模型下行为不够稳定。结果表明，GRPO为实现更透明、可信的模型推理提供了有前景的方向。
原文摘要 · Abstract (English)
Chain-of-thought (CoT) reasoning has emerged as a powerful technique for improving the problem-solving capabilities of large language models (LLMs), particularly for tasks requiring multi-step reasoning. However, recent studies show that CoT explanations often fail to reflect the model's actual reasoning process, as models may produce coherent yet misleading justifications or modify answers without acknowledging external cues. Such discrepancies undermine the reliability of CoT-based methods for safety supervision and alignment monitoring, as models can generate plausible but deceptive rationales for incorrect answers. To better understand this limitation, we evaluate two optimization methods, Group Relative Policy Optimization (GRPO) and Direct Preference Optimization (DPO), in their ability to improve CoT faithfulness. Our experiments show that GRPO achieves higher performance than DPO in larger models, with the Qwen2.5-14B-Instruct model attaining the best results across all evaluation metrics. Both approaches exhibit positive correlations between model size and performance, but GRPO shows greater potential for improving faithfulness metrics, albeit with less stable behavior at smaller scales. These results suggest that GRPO offers a promising direction for developing more transparent and trustworthy reasoning in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。