让大模型在道德难题中表现更像人,通过控制不确定性提升决策一致性。
Dropouts in Confidence: Moral Uncertainty in Human-LLM Alignment
- 用推理时的随机丢弃(dropout)引入不确定性,调节模型信心。
- 该方法显著提升模型与人类在道德判断上的对齐度,相关性明显上升。
- 适用于需高可信道德决策的场景,如医疗或司法辅助系统。
人类在面对道德困境时表现出显著不确定性,但机器与人工智能代理的这种不确定性尚未被充分研究。近期研究表明,大型语言模型(LLMs)在生成回答时往往过于自信。随着这些系统越来越多地用于伦理决策场景,理解其道德推理及内在不确定性至关重要。本研究以经典的电车难题为框架,分析32个开源模型在9个不同道德维度上的表现。结果发现,模型间信心差异远大于同一模型在不同道德维度间的差异,表明道德不确定性主要由模型架构和训练方法决定。为量化不确定性,我们采用二元熵作为总熵、条件熵与互信息的线性组合。通过推理时引入‘丢弃’(dropout)机制施加随机性,结果显示该方法显著提升总熵,主要源于互信息增加,而条件熵基本不变。更重要的是,该机制显著改善了人类与模型在道德判断上的对齐度,互信息与对齐分数均发生显著变化。结果表明,通过有意识地调控不确定性并降低大模型在复杂道德情境中的自信程度,可有效提升其决策与人类偏好的一致性。
原文摘要 · Abstract (English)
Humans display significant uncertainty when confronted with moral dilemmas, yet the extent of such uncertainty in machines and AI agents remains underexplored. Recent studies have confirmed the overly confident tendencies of machine-generated responses, particularly in large language models (LLMs). As these systems are increasingly embedded in ethical decision-making scenarios, it is important to understand their moral reasoning and the inherent uncertainties in building reliable AI systems. This work examines how uncertainty influences moral decisions in the classical trolley problem, analyzing responses from 32 open-source models and 9 distinct moral dimensions. We first find that variance in model confidence is greater across models than within moral dimensions, suggesting that moral uncertainty is predominantly shaped by model architecture and training method. To quantify uncertainty, we measure binary entropy as a linear combination of total entropy, conditional entropy, and mutual information. To examine its effects, we introduce stochasticity into models via "dropout" at inference time. Our findings show that our mechanism increases total entropy, mainly through a rise in mutual information, while conditional entropy remains largely unchanged. Moreover, this mechanism significantly improves human-LLM moral alignment, with correlations in mutual information and alignment score shifts. Our results highlight the potential to better align model-generated decisions and human preferences by deliberately modulating uncertainty and reducing LLMs' confidence in morally complex scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。