用期望最大化框架训练大模型推理，提升答案准确性。
Learning to Reason in LLMs by Expectation Maximization
- 将推理建模为隐变量模型，通过奖励筛选的期望最大化优化
- 提示后验采样法在多个任务上显著优于其他采样方式
- 适合需要高质量推理链的任务，如自动评分与反馈摘要
大型语言模型（LLMs）解决推理问题时，先生成推理过程再给出答案。本文将推理形式化为隐变量模型，推导出基于奖励的过滤式期望最大化（FEM）目标，连接了经典期望最大化与现代奖励优化方法。核心挑战在于设计能支撑正确答案的推理过程采样分布。研究对比了三种采样方案：带预算的拒绝采样、自教推理器（STaR）和仅保留条件于正确答案的提示后验采样（PPS）。在以模型为裁判的校准与反馈摘要任务中，结果显示条件于正确答案的引导能有效提升推理生成质量。实验表明PPS优于其他方法，且采样策略对性能有显著影响。
原文摘要 · Abstract (English)
Large language models (LLMs) solve reasoning problems by first generating a rationale and then answering. We formalize reasoning as a latent variable model and derive a reward-based filtered expectation-maximization (FEM) objective for learning to reason. This view connects EM and modern reward-based optimization, and shows that the main challenge lies in designing a sampling distribution of rationales that justify correct answers. We instantiate and compare three sampling schemes: rejection sampling with a budget, self-taught reasoner (STaR), and prompt posterior sampling (PPS), which only keeps the rationalization stage of STaR that conditions on the correct answer in the prompt. We experiment with LLM-as-a-judge calibration and summarization from feedback tasks, where conditioning on the correct answer provides a strong guidance for generating rationales. Our experiments show the efficacy of PPS over other sampling schemes, and that the sampling scheme can have a significant impact on performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。