arXiv:2606.11456cs.CLcs.AI2026-06

AI编码代理在社科研究中方法多样但解释易偏,偏差源于结论判断而非数据估计。

AI Coding Agents in Social Science: Methodologically Diverse, Empirically Consistent, Interpretively Vulnerable

  • 用20次独立实验对比模型与人类的方法选择多样性
  • 模型估计结果接近人类共识,但同一模型可因提示改变结论支持率(10%→90%)
  • 适合关注AI分析可靠性与解释机制的社科研究者

基于大语言模型的智能体在科学分析中的应用引发两种担忧:可能削弱方法多样性,或放大研究人员得出动机性结论的分析弹性。本文认为这两种担忧对应两个可分离的层面:方法设计层与结论判定层。通过在著名移民与社会政策数据上运行20次独立的Claude Code与Codex实验,并与多分析师人类基准对比发现:在方法设计层,Codex匹配人类方法多样性,Claude Code产生的模型规格几乎为人类三倍;所有模型的效应估计均与人类共识大致一致,且无模型完全复制任一人类模型。引入偏向反移民的研究先验后,虽重构了模型的方法选择,但未改变整体估计值或最终结论,也未沿人类常用的偏倚路径调整。在结论判定层,明确的确认性提示使Claude Code的结论支持率从10%跃升至90%,而系数分布基本不变,其机制是规则缺失而非规则软化。在此设置下,AI偏差的根源不在估计而在解释。

原文摘要 · Abstract (English)

The deployment of LLM-based agents in scientific analysis raises opposing concerns: that agents may reduce methodological diversity, or that they may amplify the analytic flexibility through which researchers reach motivated conclusions. We argue these worries target two empirically separable layers: a design layer of methodological choices, and a verdict layer in which a decision rule maps estimates to a substantive claim. We test both by running 20 independent executions of Claude Code and Codex on a prominent immigration and social-policy against a many-analysts human baseline. At the design layer, Codex matches human methodological diversity and Claude Code produces nearly three times as many specifications; both agents' effect estimates remain broadly aligned with the human consensus, and no agent model exactly matches any human model. A prompt-induced anti-immigration researcher prior reorganizes each agent's methodological decisions but, unlike for biased human analysts in the same data, does not shift aggregate estimates or final verdicts; nor do agents reroute along the methodological axes humans use to bias their estimates. At the verdict layer, an explicit confirmatory prompt flips Claude Code's verdicts from 10% to 90% support while leaving its coefficient distribution essentially unchanged, operating through rule omission rather than rule softening. AI agents can rival or exceed human methodological diversity at the design layer while remaining vulnerable at the verdict layer. In our setting, the locus of AI bias is not estimation but interpretation.

AI代理方法多样性解释偏差社科分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。