让大模型同时满足多个用户目标，还能在最差情况下保持稳定表现。
Robust Multi-Objective Controlled Decoding of Large Language Models
- 设计对抗性奖励权重与采样策略的博弈机制，确保多目标鲁棒对齐。
- 在10个目标下实验，显著提升最差情况下的奖励得分和胜率。
- 计算开销极低，适合部署到现有大模型系统中使用。
我们提出一种新的推理阶段算法——鲁棒多目标解码（RMOD），通过最大化最差情况下的奖励，实现大语言模型（LLMs）对多个人类目标（如指令遵循、有用性、安全性）的鲁棒对齐。RMOD将鲁棒解码建模为对抗性奖励权重与采样策略之间的最大最小二人博弈，可通过纳什均衡求解。该博弈可简化为凸优化问题以确定最坏情况下的奖励权重，最优采样策略可解析推导得出。针对实际应用，我们设计了适用于当前主流大模型的高效算法，相较于标准非鲁棒控制解码方法，计算开销几乎无增加。在包含多达10个目标的多个主流对齐数据集上进行实验，结果表明RMOD及其精简版本在最差情况奖励和胜率上持续优于基线方法。
原文摘要 · Abstract (English)
We introduce Robust Multi-Objective Decoding (RMOD), a novel inference-time algorithm that robustly aligns Large Language Models (LLMs) to multiple human objectives (e.g., instruction-following, helpfulness, safety) by maximizing the worst-case rewards. RMOD formulates the robust decoding problem as a maximin two-player game between adversarially computed reward weights and the sampling policy, solvable through a Nash equilibrium. We demonstrate that this game reduces to a convex optimization problem to identify the worst-case reward weights, with the optimal sampling policy analytically derived. For practical applications, we propose an efficient algorithm of RMOD tailored for contemporary LLMs, introducing minimal computational overhead compared to standard non-robust Controlled Decoding methods. Experimental results across a range of popular alignment datasets with up to 10 objectives show the effectiveness of RMOD and its distilled version, consistently outperforming baselines in worst-case rewards and win rates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。