让大模型按需切换不同观点，用思维链实现可调控的多元立场对齐。
Exploring Chain-of-Thought Reasoning for Steerable Pluralistic Alignment
- 用思维链提示与强化学习结合，训练模型根据指令切换立场。
- 强化学习方法在多个数据集上表现最优，且只需少量训练样本。
- 适合需要多角度推理、避免单一价值倾向的应用场景。
大型语言模型通常被训练为反映相对统一的价值观，限制了其在需要理解细微人类视角任务中的应用。近期研究强调了让模型具备可调控多元主义能力的重要性——即能够采纳特定视角并使生成内容与其对齐。本文探讨了将思维链（Chain-of-Thought, CoT）推理技术应用于构建可调控多元主义模型的可行性。我们考察了多种方法：思维链提示、基于人类撰写的思维链微调、基于合成解释的微调，以及基于可验证奖励的强化学习（RLVR）。在Value Kaleidoscope和OpinionQA两个数据集上的评估表明，RLVR方法始终优于其他方法，并展现出优异的训练样本效率。此外，我们还分析了生成的思维链轨迹在忠实度与安全性方面的表现。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are typically trained to reflect a relatively uniform set of values, which limits their applicability to tasks that require understanding of nuanced human perspectives. Recent research has underscored the importance of enabling LLMs to support steerable pluralism -- the capacity to adopt a specific perspective and align generated outputs with it. In this work, we investigate whether Chain-of-Thought (CoT) reasoning techniques can be applied to building steerable pluralistic models. We explore several methods, including CoT prompting, fine-tuning on human-authored CoT, fine-tuning on synthetic explanations, and Reinforcement Learning with Verifiable Rewards (RLVR). We evaluate these approaches using the Value Kaleidoscope and OpinionQA datasets. Among the methods studied, RLVR consistently outperforms others and demonstrates strong training sample efficiency. We further analyze the generated CoT traces with respect to faithfulness and safety.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。