用少量偏好数据实现大模型行为对齐，无需训练即可插件式使用。
Toward Preference-aligned Large Language Models via Residual-based Model Steering
- 基于残差流提取轻量级引导向量，推理时动态调整模型输出。
- 仅需100组偏好数据，数学与代码生成性能提升且通用能力不退化。
- 相比传统方法快数倍，适合快速部署与多场景适配。
偏好对齐是使大语言模型实用并符合人类偏好的关键步骤。现有方法如基于人类反馈的强化学习或直接偏好优化通常需要精心构建的数据集和对数十亿参数的昂贵优化，最终产生任务特定的模型。本文提出通过残差流进行大语言模型偏好对齐（PaLRS），一种无需训练的方法，利用大模型残差流中编码的偏好信号。仅需100组偏好对齐数据，PaLRS即可提取轻量、即插即用的引导向量，可在推理阶段应用以推动模型朝偏好行为演进。我们在多种中小型开源LLM上评估了PaLRS，结果显示其在数学推理和代码生成基准上持续取得提升，同时保持基线通用性能。相较于经DPO和SimPO对齐的模型，其表现更优且节省大量时间。结果表明，PaLRS提供了一种高效、灵活且无需训练的偏好对齐替代方案，仅用极少数据即可实现精准可控的模型行为调整。
原文摘要 · Abstract (English)
Preference alignment is a critical step in making Large Language Models (LLMs) useful and aligned with (human) preferences. Existing approaches such as Reinforcement Learning from Human Feedback or Direct Preference Optimization typically require curated data and expensive optimization over billions of parameters, and eventually lead to persistent task-specific models. In this work, we introduce Preference alignment of Large Language Models via Residual Steering (PaLRS), a training-free method that exploits preference signals encoded in the residual streams of LLMs. From as few as one hundred preference pairs, PaLRS extracts lightweight, plug-and-play steering vectors that can be applied at inference time to push models toward preferred behaviors. We evaluate PaLRS on various small-to-medium-scale open-source LLMs, showing that PaLRS-aligned models achieve consistent gains on mathematical reasoning and code generation benchmarks while preserving baseline general-purpose performance. Moreover, when compared to models aligned with DPO and SimPO, they perform better with great time-savings. Our findings highlight that PaLRS offers an effective, much more efficient and flexible alternative to standard preference optimization pipelines, offering a training-free, plug-and-play mechanism for alignment with minimal data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。