通过连续空间梯度优化，实现推理时高效对齐大模型与人类反馈。
Inference-time Alignment in Continuous Space
- 在连续潜空间中用梯度采样直接优化原始输出，避免离散搜索
- 在AdvBench上相对提升77.51%，在MATH上提升16.36%
- 方法简单有效，适合需要快速推理对齐的场景
由于灵活性高，推理时基于人类反馈对齐大型语言模型受到越来越多关注。现有方法依赖从基础策略生成多个响应，并使用奖励模型进行搜索，相当于在离散响应空间中搜索。然而，当基础策略较弱或候选集较小时，这些方法难以探索有信息量的候选，导致效果受限。本文提出简单能量适应（SEA），一种简单而有效的推理时对齐算法。与昂贵的离散空间搜索不同，SEA通过连续潜空间中的梯度采样，直接将基础策略的原始响应向最优解调整。具体而言，SEA将推理建模为在由最优策略定义的连续空间中对动作的能量函数进行迭代优化，实现简单而有效的对齐。例如，尽管结构简单，SEA在AdvBench上相对次优基线提升高达77.51%，在MATH上提升16.36%。代码已公开于https://github.com/yuanyige/sea。
原文摘要 · Abstract (English)
Aligning large language models with human feedback at inference time has received increasing attention due to its flexibility. Existing methods rely on generating multiple responses from the base policy for search using a reward model, which can be considered as searching in a discrete response space. However, these methods struggle to explore informative candidates when the base policy is weak or the candidate set is small, resulting in limited effectiveness. In this paper, to address this problem, we propose Simple Energy Adaptation ($\textbf{SEA}$), a simple yet effective algorithm for inference-time alignment. In contrast to expensive search over the discrete space, SEA directly adapts original responses from the base policy toward the optimal one via gradient-based sampling in continuous latent space. Specifically, SEA formulates inference as an iterative optimization procedure on an energy function over actions in the continuous space defined by the optimal policy, enabling simple and effective alignment. For instance, despite its simplicity, SEA outperforms the second-best baseline with a relative improvement of up to $ \textbf{77.51%}$ on AdvBench and $\textbf{16.36%}$ on MATH. Our code is publicly available at https://github.com/yuanyige/sea
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。