让大模型在推理时低成本适配多种人类偏好
Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPO
- 训练基础模型，使其可被不同推理对齐算法灵活调用
- 采用非线性GRPO优化，确保收敛到最优对齐解
- 适合需要多目标、低延迟推理的场景
将大语言模型（LLMs）对齐于多样化的用户偏好本质上具有挑战性，因为标准之间常存在冲突。近期，推理时对齐方法因其可在推理阶段通过不同对齐算法实现多标准适配而受到关注。然而，此类方法计算开销大，通常需多次前向传播基础模型。本文提出推理感知元对齐（IAMA），一种新方法，使模型在推理时以有限计算预算适配多种标准。IAMA 训练基础模型，使其能通过不同的推理时对齐算法有效适配多个任务。为解决 IAMA 中涉及的非线性优化问题，我们提出非线性 GRPO，该方法在概率测度空间中可证明收敛至最优解。
原文摘要 · Abstract (English)
Aligning large language models (LLMs) to diverse human preferences is fundamentally challenging since criteria can often conflict with each other. Inference-time alignment methods have recently gained popularity as they allow LLMs to be aligned to multiple criteria via different alignment algorithms at inference time. However, inference-time alignment is computationally expensive since it often requires multiple forward passes of the base model. In this work, we propose inference-aware meta-alignment (IAMA), a novel approach that enables LLMs to be aligned to multiple criteria with limited computational budget at inference time. IAMA trains a base model such that it can be effectively aligned to multiple tasks via different inference-time alignment algorithms. To solve the non-linear optimization problems involved in IAMA, we propose non-linear GRPO, which provably converges to the optimal solution in the space of probability measures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。