用偏好数据优化大模型输出，推理时更精准可控。
Pref-CTRL: Preference Driven LLM Alignment using Representation Editing
- 基于人类偏好构建多目标价值函数，指导模型生成。
- 在两个基准数据集上优于现有方法，跨领域泛化更强。
- 适合需要精准控制输出风格的研究者和开发者。
测试时对齐方法通过在推理阶段对大语言模型的内部表示进行轻量干预，提供了替代微调的可行路径。近期，一种有效的方法RE-Control（Kong等，2024）利用在模型隐藏状态上训练的外部价值函数，通过梯度编辑引导生成。然而该方法忽略了对齐任务的核心特征——通常基于候选回复间的偏好数据进行学习。为此，本文提出新的偏好驱动训练框架Pref-CTRL，采用多目标价值函数更准确地反映偏好数据结构。实验表明，该方法在两个基准数据集上优于RE-Control，且在跨域数据集上展现出更强泛化能力。代码已开源。
原文摘要 · Abstract (English)
Test-time alignment methods offer a promising alternative to fine-tuning by steering the outputs of large language models (LLMs) at inference time with lightweight interventions on their internal representations. Recently, a prominent and effective approach, RE-Control (Kong et al., 2024), has proposed leveraging an external value function trained over the LLM's hidden states to guide generation via gradient-based editing. While effective, this method overlooks a key characteristic of alignment tasks, i.e. that they are typically formulated as learning from human preferences between candidate responses. To address this, in this paper we propose a novel preference-based training framework, Pref-CTRL, that uses a multi-objective value function to better reflect the structure of preference data. Our approach has outperformed RE-Control on two benchmark datasets and showed greater generalization on out-of-domain datasets. Our source code is available at https://github.com/UTS-nlPUG/pref-ctrl.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。