arXiv:2510.18526cs.AIcs.LG2025-10NeurIPS被引 6

用反事实推理让大模型灵活适配多元价值观,提升对复杂价值优先级的控制能力。

Counterfactual Reasoning for Steerable Pluralistic Value Alignment of Large Language Models

  • 构建因果模型显式建模价值间的依赖与优先级关系
  • 在多个数据集上实现更精准的价值对齐,尤其擅长处理小众价值目标
  • 方法可解释性强,适合需要文化敏感性对齐的多场景应用

随着大语言模型日益服务于跨文化、跨社群的用户群体,将其对齐于超越平均原则(如HHH)的多元人类价值观变得至关重要。心理学与社会价值理论(如施瓦茨价值观理论)指出,多元价值观由多个价值维度及其优先级组合构成。然而现有方法面临两大挑战:一是将多个价值视为独立且同等重要,忽视其相互依赖与相对优先级(价值复杂性);二是难以精确控制细微的价值优先级,尤其是代表性不足的价值目标(价值可控性)。为此,我们提出COUPLE——一种用于多元价值对齐的反事实推理框架。该框架引入结构因果模型(SCM),刻画特征间的复杂依赖关系及高层价值维度与行为之间的因果关系。同时,通过反事实推理生成符合任意期望价值目标的输出。得益于显式的因果建模,COUPLE具备更强的可解释性。我们在两个具有不同价值体系的数据集上评估了该方法,结果表明,相比其他基线,COUPLE在多种类型的价值目标下均表现更优。

原文摘要 · Abstract (English)

As large language models (LLMs) become increasingly integrated into applications serving users across diverse cultures, communities and demographics, it is critical to align LLMs with pluralistic human values beyond average principles (e.g., HHH). In psychological and social value theories such as Schwartz's Value Theory, pluralistic values are represented by multiple value dimensions paired with various priorities. However, existing methods encounter two challenges when aligning with such fine-grained value objectives: 1) they often treat multiple values as independent and equally important, ignoring their interdependence and relative priorities (value complexity); 2) they struggle to precisely control nuanced value priorities, especially those underrepresented ones (value steerability). To handle these challenges, we propose COUPLE, a COUnterfactual reasoning framework for PLuralistic valuE alignment. It introduces a structural causal model (SCM) to feature complex interdependency and prioritization among features, as well as the causal relationship between high-level value dimensions and behaviors. Moreover, it applies counterfactual reasoning to generate outputs aligned with any desired value objectives. Benefitting from explicit causal modeling, COUPLE also provides better interpretability. We evaluate COUPLE on two datasets with different value systems and demonstrate that COUPLE advances other baselines across diverse types of value objectives.

大模型对齐反事实推理多元价值因果建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。