PPO-HSC让大模型在微调中更敢探索，避免陷入固定解法。
PPO-HSC: An Exploratory Reinforcement Learning Framework Based on Wide-Area Policy Coverage Optimization
- 引入高阶采样覆盖率奖励，鼓励发现新颖但合理的解题思路。
- 在GSM8K和SVAMP上提升解法多样性，准确率不降反升。
- 适合需要多样化推理的数学与代码生成任务研究者。
本文提出PPO-HSC（基于高阶采样覆盖率的近端策略优化），一种用于解决大语言模型微调中模式崩溃问题的探索性强化学习框架。标准验证奖励强化学习（RLVR）虽能强化高奖励路径，却常导致模型过度优化已知解法，丧失探索能力。PPO-HSC通过引入高阶采样覆盖率（HSC）奖励，激励发现‘低相似度但高有效性’的推理模式。该框架维护一个动态的已验证唯一解轨迹库，提供可微信号以奖励语义新颖性，同时通过合理性约束保证结构合理性。在数学推理（GSM8K、SVAMP）和代码生成任务上的实证表明，PPO-HSC显著提升了解空间覆盖度与解法多样性，同时保持或超越当前先进强化学习基线的准确率与语法完整性。
原文摘要 · Abstract (English)
This paper introduces PPO-HSC (Proximal Policy Optimization with High-order Sampling Coverage), an exploratory reinforcement learning framework designed to address the "Invisible Shackles" of mode collapse in Large Language Model (LLM) fine-tuning. While standard Reinforcement Learning from Verifiable Rewards (RLVR) effectively reinforces high-reward trajectories, it often leads models to over-optimize known solutions, sacrificing curiosity and the ability to explore broader solution manifolds. To overcome this, PPO-HSC incorporates a High-order Sampling Coverage (HSC) reward that incentivizes the discovery of "low-similarity yet high-validity" reasoning patterns. By maintaining a dynamic trajectory library of verified unique solutions, the framework provides a differentiable signal that rewards semantic novelty while ensuring structural rationality through a plausibility constraint. Empirical evaluations on mathematical reasoning (GSM8K, SVAMP) and code generation tasks demonstrate that PPO-HSC significantly enhances solution diversity and state-space coverage while maintaining or surpassing the accuracy and syntax integrity of state-of-the-art RL baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。