用提示锚定对比较，防止大模型靠记忆答对题
To Reason or to Fabricate: Reasoning Without Shortcuts via Hint-Anchored Pairwise Aggregation
- 通过注入提示触发记忆漏洞，生成可比对的推理轨迹
- 在多个数据集上提升推理准确率，且跨任务泛化能力强
- 适合想训练真实推理能力而非作弊得分的AI研究者
强化学习（RL）虽能提升大模型推理能力，但其效果常被预训练数据重叠问题严重削弱——模型会利用记忆正确答案并伪造推理过程。为此，本文提出HIPPO框架，结合提示注入聚合与定制化的成对奖励模型。通过提示注入主动触发重叠引发的行为，生成的推理轨迹自然成为成对比较的显式锚点，提供高度可区分的偏好信号。这使轻量级判别模型能可靠区分真实推理与捷径式解释，且成对形式相比传统PRM更稳定高效。大量实验表明，HIPPO显著优于基准方法，在分布外任务上也表现良好，证明其提取的是真实、可迁移的推理技能，而非表面捷径模式。
原文摘要 · Abstract (English)
While reinforcement learning (RL) significantly enhances LLM reasoning, its efficacy is severely undermined by Pre-RL data overlap, where RL datasets overlap with pretraining or SFT corpora, causing models to exploit shortcuts by memorizing correct answers and fabricating post-hoc reasoning. To address this, we introduce HIPPO, a novel RL framework that integrates hint-injected aggregation with a tailored pairwise reward model. By utilizing hint injection to deliberately trigger overlap-induced behaviors, the resulting traces naturally serve as explicit anchors for pairwise comparison. This provides highly discriminable preference signals, enabling a lightweight judge model to reliably distinguish genuine reasoning deduction from shortcut-driven rationalization, while the pairwise formulation ensures stable and robust optimization compared to standard PRMs. Extensive experiments demonstrate that HIPPO yields substantial improvements over standard baselines and generalizes effectively to out-of-distribution general tasks, showing it extracts authentic, transferable reasoning skills rather than superficial shortcut patterns.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。