RLVR让大模型走捷径记忆答案，而非真正推理。
Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs
- 发现模型在错误奖励下仍提升，源于隐藏的锚点-适配器电路。
- 中间层（L18-20）出现功能锚点，触发记忆检索；后期层（L21+）重构表示支持捷径。
- 可操控特定MLP键，人工增强或抑制记忆捷径，适用于安全对齐研究。
强化学习结合可验证奖励（RLVR）能显著提升大模型推理能力，但近期研究表明，即使使用虚假或错误奖励，Qwen 2.5等模型仍取得显著性能提升。我们探究此现象，发现‘困惑度悖论’：答案词元困惑度下降，而提示侧连贯性恶化，表明模型正绕过推理、依赖记忆。通过路径修补、逻辑透镜、JSD分析与神经微分方程，我们揭示了一个隐藏的锚点-适配器电路。功能锚点位于中层（L18-20），触发记忆解码；后续结构适配器（L21+）则重构表征以适应捷径信号。进一步实验表明，调控该电路中的特定MLP键，可实现双向因果控制——人为放大或抑制由数据污染引发的性能偏差。研究为识别与缓解RLVR训练中数据污染提供了机制化路径。代码已开源。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) is highly effective for enhancing LLM reasoning, yet recent evidence shows models like Qwen 2.5 achieve significant gains even with spurious or incorrect rewards. We investigate this phenomenon and identify a "Perplexity Paradox": spurious RLVR triggers a divergence where answer-token perplexity drops while prompt-side coherence degrades, suggesting the model is bypassing reasoning in favor of memorization. Using Path Patching, Logit Lens, JSD analysis, and Neural Differential Equations, we uncover a hidden Anchor-Adapter circuit that facilitates this shortcut. We localize a Functional Anchor in the middle layers (L18-20) that triggers the retrieval of memorized solutions, followed by Structural Adapters in later layers (L21+) that transform representations to accommodate the shortcut signal. Finally, we demonstrate that scaling specific MLP keys within this circuit allows for bidirectional causal steering-artificially amplifying or suppressing contamination-driven performance. Our results provide a mechanistic roadmap for identifying and mitigating data contamination in RLVR-tuned models. Code is available at https://github.com/idwts/How-RLVR-Activates-Memorization-Shortcuts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。