通过隐式优化潜在状态,让大模型在推理时自动修正错误思维路径。
Latent Reward Steering: An Adaptive Inference-Time Framework that Implicitly Promotes Cognitive Behaviors in Reasoning LLMs

- 用稀疏自编码器的潜在表示捕捉认知行为,不依赖预设规则。
- 基于答案正确性训练奖励模型,动态识别并修正脆弱推理状态。
- 适用于多种模型和任务,能自发提升推理质量,适合研究推理机制者。
强推理不仅依赖模型知识,更取决于推理过程中认知行为的有效运用。现有方法多采用显式的行为控制,难以适应不同推理状态、任务和模型下的差异性错误与修正需求。为此,我们提出隐式奖励引导(Latent Reward Steering, LRS),一种自适应的推理时框架,通过优化携带认知行为信息的稀疏自编码器(SAE)潜在状态来促进良好认知行为。LRS 不依赖预设认知行为或其引导方向,而是通过最终答案正确性在推理轨迹上训练一个潜在奖励模型,以估计中间潜在状态的质量。推理时,奖励梯度为脆弱潜在状态提供状态特定的修正方向,同时奖励与置信度门控机制仅在奖励信号标记为脆弱的状态下进行干预。在多个推理大模型主干和基准测试上的实验表明, extsc{LRS} 持续优于各类基线,后验分析进一步显示其能隐式促进修复原始推理错误的良好认知行为。代码已公开:https://github.com/jiakanglee/Latent-Reward-Steering。
原文摘要 · Abstract (English)
Strong reasoning depends not only on model knowledge but also on how effectively cognitive behaviors are deployed during generation. Existing methods often rely on explicit behavior-level control, making them insufficiently adaptive when failures and required corrections vary across reasoning states, tasks, and models. To this end, we propose Latent Reward Steering (LRS), an adaptive inference-time framework that promotes cognitive behaviors by optimizing the sparse-autoencoder (SAE) latent states that implicitly carry them. Rather than relying on predefined cognitive behaviors or steering directions derived from them, LRS trains a latent reward model on reasoning traces by final answer correctness to estimate the quality of intermediate latent states. During inference, reward gradients provide state-specific correction directions for fragile latent states, while a reward and confidence gate restricts intervention to states the reward signal flags as fragile. Experiments on multiple reasoning LLM backbones and benchmarks show that \ours consistently improves performance over various baselines, and post-hoc analyses further indicate that \ours implicitly promotes good cognitive behaviors that fix the original reasoning errors. Code is available at: https://github.com/jiakanglee/Latent-Reward-Steering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。