让大模型根据评分标准动态调整输出风格，无需重新训练。
Configurable Preference Tuning with Rubric-Guided Synthetic Data
- 用结构化评分标准生成合成偏好数据，指导模型学习可配置行为。
- 模型在推理时根据系统提示自动调整输出，实现细粒度风格控制。
- 适合需要灵活适配不同写作风格的场景，如内容创作与合规生成。
当前基于人类反馈的对齐模型（如直接偏好优化）通常采用单一、静态的偏好设定，限制了灵活性。本文提出可配置偏好调优（CPT）框架，使语言模型能够根据明确的人类可读指令动态调整行为。CPT利用基于结构化细粒度评分标准生成的合成偏好数据，通过条件化系统提示来定义期望属性（如写作风格）。在该数据上微调后，大模型可在推理阶段响应系统提示，无需重新训练即可调节输出。该方法不仅实现精细控制，还支持建模更复杂、情境相关的偏好。相关代码、生成数据集及微调模型已开源：https://github.com/vicgalle/configurable-preference-tuning。
原文摘要 · Abstract (English)
Models of human feedback for AI alignment, such as those underpinning Direct Preference Optimization (DPO), often bake in a singular, static set of preferences, limiting adaptability. This paper challenges the assumption of monolithic preferences by introducing Configurable Preference Tuning (CPT), a novel framework for endowing language models with the ability to dynamically adjust their behavior based on explicit, human-interpretable directives. CPT leverages synthetically generated preference data, conditioned on system prompts derived from structured, fine-grained rubrics that define desired attributes like writing style. By fine-tuning with these rubric-guided preferences, the LLM learns to modulate its outputs at inference time in response to the system prompt, without retraining. This approach not only offers fine-grained control but also provides a mechanism for modeling more nuanced and context-dependent human feedback. Several experimental artifacts, such as training code, generated datasets and fine-tuned models are released at https://github.com/vicgalle/configurable-preference-tuning
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。