arXiv:2608.29803cs.CYcs.CL2026-08

LLMs不像人类那样被说服,关键差异在认知机制。

Do LLMs Change Their Minds Like Humans? Diagnosing Human--LLM Divergence in Single-Turn Persuasion Judgments

论文配图:Do LLMs Change Their Minds Like Humans? Diagnosing Human--LLM Divergence in Single-Turn Persuasion Judgments
图 1 · 摘自论文原文
  • 用真实在线辩论数据对比人类与LLM的信念更新
  • 两者仅微弱一致(kappa 0.079~0.178),对细微说服信号反应不同
  • 适合研究人机认知差异或评估模型社会模拟可靠性

大型语言模型(LLMs)正越来越多地被用作社会模拟中的人类参与者替代品,但它们是否像人类一样会根据说服性论据更新自身信念仍不明确。本研究利用一个自然发生的在线说服语料库,其中原始发帖者明确验证回复是否改变了其观点。结果显示,LLMs与人类的判断一致性极低(Cohen's kappa 0.079 至 0.178)。内容层面分析表明,人类和LLMs仅在最强说服线索上达成共识,但在细微差异上存在分歧:人类更易受新颖内容和强势语言影响,而LLMs则更关注主题相似性和表面格式。在说服策略层面,LLMs低估情感诉求、高估可信度信号,而争议议题类型对分歧程度无显著影响。此外,从第一人称角色扮演切换为第三人称观察,使所有模型均表现出更强的抗拒说服倾向,且该效应因说服策略和文本特征而异。这些发现揭示了将LLM判断视为人类信念更新忠实代理的风险,并指出了人机在处理说服性话语上的结构性差异。代码已公开于 https://github.com/tsinghua-fib-lab/LLM-belief-update-cmv。

原文摘要 · Abstract (English)

Large language models (LLMs) are increasingly deployed as proxies for human participants in social simulations, yet whether they update their beliefs in response to persuasive arguments, as humans do, remains poorly understood. We conduct a systematic comparison using a naturally occurring online persuasion corpus in which original posters explicitly verify whether a reply changed their view. Our results show that LLMs achieve only slight agreement with humans (Cohen's kappa ranging from 0.079 to 0.178). Content-level analyses show that humans and LLMs agree on the strongest persuasion cues but diverge on finer ones: humans are more swayed by novel content and assertive language, whereas LLMs favor topical similarity and surface-level formatting. At the level of persuasion strategy, LLMs underweight emotional appeals and overweight credibility signals relative to humans, while the type of proposition under debate exerts no measurable effect on the degree of divergence. Furthermore, switching from first-person role-playing to third-person observation shifts all models toward greater resistance to persuasion, with the effect varying across persuasion strategies and textual features. These findings highlight the risk of treating LLM judgments as faithful proxies for human belief updating and point to structural differences in how LLMs and humans process persuasive discourse. Our code is available at https://github.com/tsinghua-fib-lab/LLM-belief-update-cmv.

大模型认知说服力分析人机差异

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。