用策略梯度实现模型行为的可逆动态调整
Policy Gradient Steering: Interventions from Behavioral Objectives
- 将行为调控建模为强化学习问题，通过少量演示积累梯度生成可移除任务向量
- 在双路径网格世界中验证了调控的校准性与可逆性，棋局中多目标可叠加生效
- 适用于足球等复杂场景，能改变团队行为且跨对手有效，适合需要灵活适应的系统
激活调控已成为大语言模型中一种轻量级方法，可在推理时动态改变模型行为。然而我们发现，现有调控方法在简单的双路径网格世界环境中也无法有效引导策略。为此，我们提出策略梯度调控（PGS），将调控过程建模为强化学习问题。PGS通过少量轨迹或示范累积临时行为目标的梯度，构建一个可移除的任务向量。我们首先在双路径网格世界中验证了PGS的校准性与可逆性。接着，在国际象棋谜题中独立训练多个PGS向量，分别测试其单独作用及组合效果，发现兼容的战术目标可协同增强。最后，在对抗性足球环境中，我们展示了PGS能够改变特定团队行为，且其影响可跨对手迁移。这些结果表明，策略梯度为多样决策领域中的临时、可组合行为适配提供了一个自然接口。
原文摘要 · Abstract (English)
Activation steering has emerged in large language models as a lightweight alternative for dynamically changing a model's behavior at inference time. However, we show that existing steering methods fail to steer even a simple policy in a two-route gridworld environment. To address this limitation, we propose Policy Gradient Steering (PGS), which formulates steering as a reinforcement learning problem. PGS accumulates gradients of a temporary behavioral objective over a small set of rollouts or demonstrations to construct a removable task vector. We first demonstrate the calibration and reversibility of PGS in a two-route gridworld environment. Using chess puzzles, we then evaluate independently fitted PGS vectors both in isolation and in combination, finding that compatible tactical objectives accumulate constructively. Finally, in competitive football, we show that PGS can alter specific team behaviors and that its effects transfer across opponents. Together, these results show that policy gradients provide a natural interface for constructing temporary and composable behavioral adaptations across diverse decision-making domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。