研究发现大模型只能说服心理易受控的人，靠信任和情绪，而非逻辑。
LLMs can persuade only psychologically susceptible humans on societal issues, via trust in AI and emotional appeals, amid logical fallacies
- 通过长期对话实验，量化大模型在社会议题上的说服力
- 44%的感知人性化由人口与心理特征决定，非观点变化
- 高信任度、外向、爱思考者更易被大模型说服
缺乏纵向证据揭示大模型在随时间演变的心理框架下对人类的说服力与类人感。我们提出Talk2AI，一个纵向框架，量化大模型在极化社会议题上的心理-社会、推理与情感维度说服力。4组纵向实验中，770名参与者与四个领先大模型就气候变化、社交媒体谣言、数学焦虑等话题展开结构化对话，共产生3,080次对话，累计60,000轮交流。每轮后，参与者报告初始立场信念强度、感知意见变化、大模型类人感、自我捐赠及文本解释。反馈时序显示信念惯性，表明即使反复接触AI论点，部分人仍锚定初始立场。有趣的是，自然语言分析显示，人类与大模型每6轮对话中均出现1次逻辑谬误，反驳了“大模型认知更优”的刻板印象。大模型类人感可由社会人口学、心理与参与特征解释（R²=0.44），其次为意见变化（R²=0.34）、信念强度（R²=0.26）和个人捐赠（R²=0.24）。可解释AI（XAI）表明：（i）存在更易被大模型影响的个体；（ii）心理易感性表现为更高大模型信任度、更高宜人性与外向性，以及更高认知需求。多宇宙混合效应模型验证了这些结果，并凸显显著个体差异。Talk2AI为生成式AI如何通过多重心理社会路径影响人类意见提供了实证基础与检测框架。
原文摘要 · Abstract (English)
Scarce longitudinal evidence examines LLMs' persuasiveness and humanness along time-evolving psychological frameworks. We introduce Talk2AI, a longitudinal framework quantifying psycho-social, reasoning and affective dimensions of LLMs' persuasiveness about polarizing societal topics. In a four-way longitudinal setup, Talk2AI's 770 participants engaged in structured conversations with one of four leading LLMs on topics like climate change, social media misinformation, and math anxiety. This produced 3,080 conversations over 60,000 turns. After each wave, participants reported conviction in their initial topic stance, perceived opinion change, LLM's perceived humanness, a self-donation to the topic and a textual explanation. Feedback time series showed longitudinal inertia in convictions, indicating some human anchoring to initial opinions even after repeated exposure to AI-generated arguments. Interestingly, NLP analyses revealed that both humans and LLMs relied on fallacious reasoning in 1 conversational quip every 6, countering the ``LLMs as superior systems" stereotype behind LLMs' cognitive surrender. LLMs' perceived humanness was most learnable from sociodemographic, psychological and engagement features ($R^2=0.44$), followed by opinion change ($R^2=0.34$), conviction ($R^2=0.26$) and personal endowment ($R^2=0.24$). Crucially, explainable AI (XAI) indicated: (i) the presence of individuals more susceptible to LLM-based opinion changes; (ii) psychological susceptibility to LLM-convincing consisted of having more trust in LLMs, being more agreeable and extraverted and with a higher need for cognition. A multiverse approach with mixed-effects models confirmed XAI results, alongside strong individual differences. Talk2AI provides a grounded framework and evidence for detecting how GenAI can influence human opinions via multiple psycho-social pathways in AI-human digital platforms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。