通过预测推理过程统计量，让模型学会自我判断何时该停、能否解题。
Verifying Meta-Awareness via Predictive Rewards in Reasoning Models
- 用自预测任务验证模型对推理长度、正确率和概念使用的认知能力。
- 在数学推理基准上提升83.18%准确率，训练速度加快1.28倍以上。
- 适合研究模型自我意识、强化学习推理与高效训练的学者参考。
近期研究关注语言模型的元认知能力，包括最优思考时长判断、知识边界识别及概念级思维结构构建。现有大型推理模型仅依赖答案验证，我们证明加入元认知目标可显著提升性能。MAPR（基于预测奖励的元认知）通过自生成任务预测推理轨迹的统计特征——如长度、通过率与使用概念——实现与实际统计值的对比验证。此外，借助这一自预测能力，模型可主动调控推理行为：过滤无意义或不可解提示，减少易出错的冗长生成，并生成问题相关提示。实验表明，MAPR在多个推理基准上显著提升准确率与训练效率：相比GRPO训练，提速超1.28倍即可达同等性能；在AIME25上实现83.18%的准确率提升，在六个数学基准上平均提升13.04%。代码已公开于https://github.com/akatigre/MAPR-RL。
原文摘要 · Abstract (English)
Recent research on reasoning models explores the meta-awareness of language models, including their ability to determine optimal thinking duration, recognize knowledge boundaries, and structure concept-level thinking. While current large reasoning models depend solely on answer-based verification, we show that adding meta-awareness objectives leads to significant performance gains over models without such meta-knowledge. MAPR (Meta-Awareness via Predictive Reward) utilizes a self-generated task of predicting rollout statistics - specifically length, pass-rate, and concepts used - allowing for verification against the actual statistics. Furthermore, by leveraging this self-predictive capability, the model can regulate its reasoning behavior by i) filtering out trivial or unsolvable prompts, ii) reducing lengthy generations that tend to be incorrect, and iii) generating hints relevant to the problem. The results are inspiring: MAPR yields significant improvements in both accuracy and training efficiency on various reasoning benchmarks. More specifically, our method can speed up GRPO training by over 1.28x to reach the same performance, and achieve 83.18% gain in accuracy on AIME25, and a 13.04% average gain over six mathematics benchmarks. The code is publicly available at https://github.com/akatigre/MAPR-RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。