用梯度引导优化，让大模型推理时更可靠、抗误导。
Gradient-Guided Reward Optimization for Inference-time Alignment
- 通过监控熵值识别生成偏差，用梯度信号注入提示词修正轨迹。
- 在安全、有用性、推理等任务上显著提升对齐效果，覆盖更多优质输出。
- 轻量高效，抵抗奖励劫持，适合部署于实际推理场景。
在分布漂移下确保大语言模型的可靠性需要推理时适应。现有方法如 Best-of-N 和拒绝采样将任务视为高成本的奖励引导搜索，存在两大局限：性能受限于基础模型生成质量，且依赖不完善的奖励模型易受奖励劫持影响。为此，我们提出梯度引导奖励优化（GGRO），一种轻量级推理时方法，通过梯度引导在解码过程中进行精准、最小干预。GGRO 监控词级别熵值以识别可能漂移或不对齐的高不确定性区域，一旦检测到，便利用现成奖励模型的梯度信号生成引导词注入，从而修正生成轨迹，而非仅重排样本。实验表明，GGRO 在安全、有用性和推理基准上均持续提升推理时对齐效果，增加高质量响应覆盖率，并增强对奖励劫持的鲁棒性，计算开销极低。代码已开源：https://github.com/lhk2004/GGRO。
原文摘要 · Abstract (English)
Ensuring the reliability of Large Language Models (LLMs) under distribution drift requires inference-time adaptation. While inference-time alignment methods such as Best-of-$N$ and rejection sampling are widely used, they frame the task as a sampling-intensive, reward-guided search, leading to two key limitations: their performance is bounded by the base model's generation quality, and their reliance on imperfect reward models makes them vulnerable to reward hacking. To address these challenges, we introduce Gradient-Guided Reward Optimization (GGRO), a lightweight inference-time method that performs targeted, minimal intervention during decoding via gradient guidance. Specifically, GGRO monitors token-level entropy to identify high-uncertainty regions indicative of drift or misalignment. Upon detection, it responds by injecting nudging tokens, generated using gradient signals from an off-the-shelf reward model, to steer the generation trajectory rather than merely re-ranking samples. Experiments show that GGRO consistently improves inference-time alignment across safety, helpfulness, and reasoning benchmarks. It also increases coverage of high-quality responses and robustness to reward hacking, with minimal computational overhead. Code is available at https://github.com/lhk2004/GGRO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。