PRISM让大模型在无真实答案时也能高效训练，靠的是内部信心与过程奖励模型结合。
PRISM: A Unified Framework for Post-Training LLMs Without Verifiable Rewards
- 用过程奖励模型+模型自信心联合指导训练
- 相比仅依赖内部一致性，训练更稳定、推理性能更好
- 适合无标注数据场景，尤其数学与代码生成任务
当前大模型后训练方法依赖昂贵的人工标注或外部验证器来提升数学推理和代码生成能力。随着模型能力增强，高质量难题解答越来越难获取，因此无监督学习愈发重要。现有方法通过模型一致性（如多数投票或置信度转奖励）提取信号，但内部一致性指标如熵或自确认性在大规模长期训练中不可靠。为此，我们提出PRISM框架，引入过程奖励模型（PRM）与模型内部信心协同引导训练，在无真实标签条件下实现稳定训练与更优测试表现，同时维持模型自信心的合理性。代码已开源。
原文摘要 · Abstract (English)
Current techniques for post-training Large Language Models (LLMs) rely either on costly human supervision or on external verifiers to boost performance on tasks such as mathematical reasoning and code generation. However, as LLMs improve their problem-solving, any further improvement will potentially require high-quality solutions to difficult problems that are not available to humans. As a result, learning from unlabeled data is becoming increasingly attractive in the research community. Existing methods extract learning signal from a model's consistency, either by majority voting or by converting the model's internal confidence into reward. Although internal consistency metric such as entropy or self-certainty require no human intervention, as we show in this work, these are unreliable signals for large-scale and long-term training. To address the unreliability, we propose PRISM, a unified training framework that uses a Process Reward Model (PRM) to guide learning alongside model's internal confidence in the absence of ground-truth labels. We show that effectively combining PRM with self-certainty can lead to both stable training and better test-time performance, and also keep the model's internal confidence in check. Code available at https://github.com/ghimiremukesh/PRISM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。