arXiv:2510.09388cs.LGcs.CL2025-10ACL被引 9

通过抽象引导提升大模型推理能力,避免答案泄露导致训练失效

Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts

  • 用元提示代替部分答案,引导模型自主思考
  • 在多个基准上超越强基线,且对分布外任务泛化更强
  • 提出可量化引导适配度的新指标,适合复杂推理研究者

强化学习已成为提升大语言模型长链推理能力的关键动力。然而,主流方法如GRPO在任务难度超过模型容量时,常因奖励稀疏和训练效率低下而失效。此前工作尝试通过离策略数据缓解问题,但往往引入严重分布偏移,导致策略更新不稳定。本文识别出根本原因——低训练亲和性,并提出首个量化外部引导与模型内在策略兼容性的指标Affinity。为此,我们设计HINT框架:首先,不提供部分答案,而是注入元提示作为抽象认知支架,促使模型独立生成解答;其次,引入亲和度感知策略优化(AAPO),动态调节学习目标以维持高亲和性。大量实验表明,HINT在多个基准上持续优于强基线,兼具优异稳定性与对分布外任务的鲁棒泛化能力。代码已开源。

原文摘要 · Abstract (English)

Reinforcement Learning (RL) has become a key driver for enhancing the long chain-of-thought (CoT) reasoning capabilities of Large Language Models (LLMs). However, prevalent methods like GRPO often fail when task difficulty exceeds model capacity, leading to reward sparsity and inefficient training. Prior work attempts to mitigate this with off-policy data, but such methods often induce severe distributional mismatches that destabilize policy updates. In this work, we identify a core issue underlying these failures, which we term low training affinity, and introduce Affinity, the first quantitative metric for monitoring the compatibility between external guidance and the model's intrinsic policy. To address this, we propose HINT, an adaptive framework designed to enhance reasoning capabilities while explicitly preserving high Affinity. First, instead of revealing partial answers, HINT supplies Meta-Hints, which act as abstract cognitive scaffolding to guide the model in articulating solutions independently. Second, to ensure stability, we integrate Affinity-Aware Policy Optimization (AAPO), which dynamically modulates the learning objective based on the Affinity. Extensive experiments across diverse benchmarks demonstrate that HINT consistently outperforms strong baselines, while exhibiting superior stability and robust generalization to out-of-distribution tasks. Code is available at https://github.com/ViviqwerAsd/HINT.

强化学习推理增强大模型提示工程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。