大模型应学习个人偏好而非平均偏好，更贴近真实用户需求。
Large Language Models Should Learn Personalized Rather Than Aggregated Human Preferences
- 用个体偏好替代平均偏好，避免忽视多样性与上下文依赖
- 聚合偏好会掩盖真实价值差异，影响模型个性化表现
- 提出可管控的个性化框架，兼顾个体自由与集体安全
当前对齐大语言模型的方法将多样的人类偏好聚合为单一奖励信号，实质上优化的是一个并不存在的“平均用户”。本文主张大模型应学习个性化、个体化的偏好，而非聚合偏好。我们指出，偏好聚合在理论上违背社会选择理论，在实证上也暴露于不同人群间显著差异。文章分析了人类偏好所蕴含的丰富结构，综述个性化技术路径，并系统回应规模化、共享标准与操纵风险等质疑。尽管个性化可能引发信息茧房、价值固化与心理操控等安全问题，但通过设定边界约束的个性化框架，可在保障通用安全的前提下容纳合理个体差异。最后，提出具体的研究与政策议程，推动具备偏好感知能力的模型发展，既尊重个体自主，也维护集体安全。
原文摘要 · Abstract (English)
Current approaches to aligning large language models (LLMs) aggregate diverse human preferences into a single reward signal, effectively optimizing for a hypothetical ``average user'' who represents no real person particularly well. This position paper argues that LLMs should learn personalized, individual preferences rather than aggregated ones. We show that aggregation masks critical information about preference diversity, individual values, and contextual dependencies, which is a limitation both theoretically grounded in social choice theory and empirically evident across demographic groups. We analyze the rich structure that human preferences encode, survey technical approaches to personalization, and systematically address counterarguments on scalability, shared standards, and manipulation risk. While personalization introduces genuine safety challenges including filter bubbles, value lock-in, and psychological manipulation, we argue these are manageable through bounded personalization frameworks that preserve universal safety constraints while accommodating legitimate individual variation. We conclude with a concrete research and policy agenda for developing preference-aware models that respect both individual autonomy and collective safety.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。