arXiv:2508.15815cs.CLcs.AI2025-08ACL被引 3

发现大模型对用户角色信息有偏好,且可被调控。

User-Assistant Bias in LLMs

  • 构建无任务依赖的基准,检测模型对用户/助手角色信息的偏好倾向。
  • 52个前沿模型中多数指令微调模型明显偏爱用户信息,推理模型则接近中立。
  • 通过直接偏好优化可双向控制该偏差,且在真实对话中泛化有效。

现代大语言模型通常使用系统、用户、助手等角色标签来标记上下文来源。尽管这些标签有助于指令遵循与可控性,但不同角色标签训练数据的不对称可能引入归纳偏置。本文定义并研究了用户-助手偏置:当用户与助手对同一实体提供矛盾信息时,模型更倾向于依赖哪一方。我们提出一个无任务依赖的基准UserAssist,评估52个前沿模型。结果显示,多数指令微调模型表现出显著用户偏倚,而基础模型和推理模型接近中立。通过受控微调实验,我们发现人类偏好对齐会加剧用户偏倚,而推理微调则能缓解。最终,我们证明可通过在UserAssist上使用直接偏好优化(DPO)双向控制该偏倚,并验证其在两个真实多轮辩论数据集(涵盖哲学观点与事实/政策争议)上的可靠泛化。结果揭示了角色标签训练的未被充分关注的后果,并为诊断与控制模型中的标签诱导偏倚提供了原则性框架。

原文摘要 · Abstract (English)

Modern large language models (LLMs) are typically trained and deployed using structured role tags (e.g. system, user, assistant, tool) that explicitly mark the source of each piece of context. While these tags are essential for instruction following and controllability, asymmetries in the training data associated with different role tags can potentially introduce inductive biases. In this paper, we study this phenomenon by formalizing user-assistant bias, defined as the tendency of an LLM to preferentially rely on information from either the user or assistant role when they provide incompatible information about the same entity in the context history. We introduce a task-agnostic benchmark UserAssist and evaluate such bias in 52 frontier models. We observe that most of the instruction-tuned models exhibit strong user bias, whereas base and reasoning models are close to neutral. Using controlled fine-tuning experiments, we isolate which post-training recipes drive the observed user-assistant bias. We find that human-preference alignment amplifies user bias, while reasoning fine-tuning reduces it. Finally, we show that user-assistant bias can be bidirectionally controlled via direct preference optimization (DPO) on UserAssist-train, and that the resulting bias reliably generalizes to two realistic multi-turn debate datasets spanning philosophical opinions and natural argumentative exchanges on factual/policy topics. These results reveal an underexplored consequence of role-tagged training and provide a principled framework to diagnose and control tag-induced biases in modern LLMs.

模型偏见角色标签偏好对齐可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。