智能补全推理关键步骤，让模型从依赖提示走向自主思考。
Placing Puzzle Pieces Where They Matter: A Question Augmentation Framework for Reinforcement Learning
- 按难度动态选择性注入关键推理步骤作为提示
- 1.5B模型在6个数学题集上达到32B模型水平，通过率保持稳定
- 渐进撤除提示，提升模型独立推理能力，避免过拟合
强化学习已成为提升大语言模型推理能力的有效方法，但面临根本困境：在简单问题上训练易导致过拟合和pass@k性能下降；而在难题上训练则常因奖励稀疏而难以收敛。现有问题增强方法通过添加部分解法作为提示来缓解，但均匀提供提示可能引入冗余信息，遗漏关键推理瓶颈，且过多提示会降低推理多样性，导致pass@k下降。我们提出PieceHint框架，能智能识别并针对性地提供关键推理步骤。该框架通过评估不同推理步骤的重要性，依据问题难度选择性分配提示，并逐步撤除辅助支撑，使模型实现从受引导学习到自主推理的过渡。在六个数学推理基准上的实验表明，我们的1.5B模型平均性能媲美32B基线，且在所有k值下均保持pass@k多样性。
原文摘要 · Abstract (English)
Reinforcement learning has become a powerful approach for enhancing large language model reasoning, but faces a fundamental dilemma: training on easy problems can cause overfitting and pass@k degradation, while training on hard problems often results in sparse rewards. Recent question augmentation methods address this by prepending partial solutions as hints. However, uniform hint provision may introduce redundant information while missing critical reasoning bottlenecks, and excessive hints can reduce reasoning diversity, causing pass@k degradation. We propose \textbf{PieceHint}, a hint injection framework that strategically identifies and provides critical reasoning steps during training. By scoring the importance of different reasoning steps, selectively allocating hints based on problem difficulty, and progressively withdrawing scaffolding, PieceHint enables models to transition from guided learning to independent reasoning. Experiments on six mathematical reasoning benchmarks show that our 1.5B model achieves comparable average performance to 32B baselines while preserving pass@k diversity across all $k$ values.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。