让每个用户定义自己的价值观，让AI更懂个人偏好。
Democratizing Reward Design for Personal and Representative Value-Alignment
- 通过对话引导用户反思并明确个人价值定义。
- 30人实验显示能准确捕捉个体独特价值观。
- 适合个性化对齐与公平性要求高的场景。
将AI代理与人类价值观对齐面临挑战，因价值观多样且主观。传统对齐方法常聚合众包反馈，易压制少数或独特偏好。本文提出交互式反思对话对齐方法，通过语言模型迭代引导用户反思并定义自身主观价值。该系统基于语言模型进行偏好获取，构建个性化奖励模型以对齐AI行为。我们在30名参与者中开展两项研究，分别聚焦“尊重”和自动驾驶伦理决策。结果揭示了价值观对齐行为的多样性，并证明本系统能准确捕捉个人独特理解。该方法支持个性化对齐，可为更具代表性和可解释性的集体对齐策略提供依据。
原文摘要 · Abstract (English)
Aligning AI agents with human values is challenging due to diverse and subjective notions of values. Standard alignment methods often aggregate crowd feedback, which can result in the suppression of unique or minority preferences. We introduce Interactive-Reflective Dialogue Alignment, a method that iteratively engages users in reflecting on and specifying their subjective value definitions. This system learns individual value definitions through language-model-based preference elicitation and constructs personalized reward models that can be used to align AI behaviour. We evaluated our system through two studies with 30 participants, one focusing on "respect" and the other on ethical decision-making in autonomous vehicles. Our findings demonstrate diverse definitions of value-aligned behaviour and show that our system can accurately capture each person's unique understanding. This approach enables personalized alignment and can inform more representative and interpretable collective alignment strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。