测试大模型说的和做的是否一致,发现差距普遍存在。
Mind the Value-Action Gap: Do LLMs Act in Alignment with Their Values?
- 构建14.8k条跨文化价值行为数据集,评估模型言行一致性。
- 不同模型和场景下,言行一致率普遍偏低,最高仅63%。
- 加入推理解释可提升预测能力,适合关注模型伦理风险的研究者。
现有研究主要通过考察大模型对特定价值观的陈述来评估其价值取向。然而,环境与社会心理学中的“价值-行为差距”现象表明,个体的陈述价值与其真实行为之间常存在差异。本研究探讨大模型在多大程度上也表现出类似的价值-行为不一致。为此,提出ValueActionLens评估框架,包含涵盖十二种文化、十一类社会议题的14.8k条价值引导行为数据集,以及三项任务,用于评估大模型在三种对齐方式下的陈述价值与价值驱动行为的一致性。大量实验显示,大模型的陈述价值与实际行为间对齐度不佳,且在不同场景与模型间差异显著。对不一致结果的分析揭示了潜在危害。进一步研究发现,引入有理由的解释能有效提升对价值-行为差距的预测性能。这些发现警示:仅凭模型陈述价值无法准确预测其行为,强调必须进行情境感知的价值评估与差距分析。
原文摘要 · Abstract (English)
Existing research primarily evaluates the values of LLMs by examining their stated inclinations towards specific values. However, the "Value-Action Gap," a phenomenon rooted in environmental and social psychology, reveals discrepancies between individuals' stated values and their actions in real-world contexts. To what extent do LLMs exhibit a similar gap between their stated values and their actions informed by those values? This study introduces ValueActionLens, an evaluation framework to assess the alignment between LLMs' stated values and their value-informed actions. The framework encompasses the generation of a dataset comprising 14.8k value-informed actions across twelve cultures and eleven social topics, and two tasks to evaluate how well LLMs' stated value inclinations and value-informed actions align across three different alignment measures. Extensive experiments reveal that the alignment between LLMs' stated values and actions is sub-optimal, varying significantly across scenarios and models. Analysis of misaligned results identifies potential harms from certain value-action gaps. To predict the value-action gaps, we also uncover that leveraging reasoned explanations improves performance. These findings underscore the risks of relying solely on the LLMs' stated values to predict their behaviors and emphasize the importance of context-aware evaluations of LLM values and value-action gaps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。