用少量偏好判断生成可读指令，让大模型推理时行为更精准。
Towards Spec Learning: Inference-Time Alignment from Preference Pairs
- 基于用户指令和少量偏好判断生成自然语言规格
- 在专业领域数据集上表现优于直接偏好优化（DPO）
- 规格可读透明，适合需要解释性的应用场景
引导大语言模型（LLM）实现期望行为通常依赖反复人工调试提示词，过程繁琐且易出错。基于偏好的微调虽更严谨，但成本高昂。本文提出规范学习（spec learning），仅需简短用户指令和少量偏好判断，即可生成自然语言形式的规格化提示，用于推理阶段约束模型行为，无需修改模型参数。实验表明，在偏好信号密集的专业领域数据集上，该方法生成的响应性能优于直接偏好优化（DPO）。相比隐晦的权重更新，生成的规格具有可读性，可作为偏好信号的透明书面表达，便于理解与验证。
原文摘要 · Abstract (English)
Steering a large language model (LLM) toward a desired behavior typically relies on an iterative process of hand-crafting a prompt based on a careful inspection of the model's responses. This is an involved, brittle, and error-prone process. Preference-based fine-tuning is a more rigorous but often prohibitively expensive solution. We propose spec learning, a framework that relies on a brief user instruction and a small set of preference judgments. These are compiled into specifications in the form of natural-language prompts for an LLM. Specifications condition LLMs at inference time, and no parameter updates to the underlying models are required. We show that the responses generated based on the compiled specifications often outperform direct preference optimization (DPO) on datasets from specialized domains whose preference signal is dense. Unlike opaque weight updates, the resulting specifications are human-readable and double as interpretable and transparent written embodiments of the preference signal that produced them.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。