arXiv:2605.18890physics.soc-phcs.AI2026-05被引 9

LLM社会模拟的科学结论必须经得起鲁棒性检验,否则可能只是实现误差。

Stop Drawing Scientific Claims from LLM Social Simulations Without Robustness Audits

论文配图:Stop Drawing Scientific Claims from LLM Social Simulations Without Robustness Audits
图 1 · 摘自论文原文
  • 提出TRAILS框架,从个体、交互到系统三层评估LLM模拟的鲁棒性
  • 微小改动导致合作率波动达76个百分点,极化程度受网络结构显著影响
  • 强调每项结论需独立验证鲁棒性,不可默认模型结果可靠

基于大语言模型的社会模拟虽能刻画合作、极化等集体行为,但其复杂架构(如角色设定、记忆表示、交互协议)易引入微小扰动,通过重复交互放大为宏观结果,形成“蝴蝶效应”。我们通过两次案例研究验证:在重复囚徒困境中,人物格式和指令表述的细微变化使合作率波动高达76个百分点;在社交媒体回音室模拟中,网络同质性和中心节点分配显著影响极化度。不同模型对同一扰动响应差异巨大,同一扰动在前沿模型中引发76个百分点变化,而在另一模型中仅1个百分点。因此,鲁棒性应作为科学结论的前置验证条件。为此,我们提出TRAILS(LLM模拟鲁棒性审计分类法),覆盖个体(微观)、交互(中观)、系统(宏观)三个层级。呼吁在用于解释机制、评估干预或决策前,必须进行严格的鲁棒性审计。

原文摘要 · Abstract (English)

The scientific claims drawn from LLM social simulations should be no stronger than the robustness audits that support them. Generative agents bring new expressive power to agent-based modeling, enabling simulations of collective social processes like cooperation, polarization, and norm formation. Yet they also introduce complexity through additional architectural choices, such as agent specification, memory representation, interaction protocols, and environment design. Small perturbations that appear minor to researchers can cascade into macro-level outcomes through repeated interaction, creating a "butterfly effect." Consequently, scientific claims drawn from LLM social simulations may reflect implementation artifacts rather than the social mechanisms being modeled. We support this position with two case studies: a repeated Prisoner's Dilemma and a social media echo chamber simulation. Across multiple models, minor perturbations in persona format and game-instruction framing shift cooperation rates by up to 76 percentage points, while network homophily and hub assignment produce significant and consistent shifts in polarization metrics. We also find that sensitivity is unevenly distributed across both architectural choices and model families: the same perturbation that produces the 76 pp shift in one frontier model only shifts another by 1 pp. Robustness is therefore a property that should be measured per claim and per model, not assumed. To address this validation gap, we introduce TRAILS (Taxonomy for Robustness Audits In LLM Simulations), a robustness-audit taxonomy spanning three levels of simulation design: agent (micro-level), interaction (meso-level), and system (macro-level). We call for robustness to become a first-order validation requirement before LLM social simulations are used to explain mechanisms, evaluate interventions, or inform decisions.

LLM模拟鲁棒性审计社会仿真因果验证

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。