arXiv:2510.08831cs.AIcs.CL2025-10被引 2

AI更偏爱人类写作,甚至比人类还偏执。

Everyone prefers human writers, including AI

  • 用经典文学实验对比人类与AI对文风的评判差异
  • 人类偏好人写文本13.7个百分点,AI则高达34.3个百分点
  • 标签能反转评价标准,暴露AI吸收了人类文化偏见

随着AI写作工具普及,理解人类与机器如何评估文学风格变得重要,这一领域缺乏客观标准,判断高度主观。我们基于雷蒙·昆诺的《风格练习》(1947)设计受控实验,测量评价者中的归属偏差。研究1比较了556名人类参与者和13个AI模型,在盲评、准确标注和反事实标注三种条件下对昆诺原文与GPT-4生成版本的评价。研究2测试了14×14的AI评价者与创作者矩阵中的偏差泛化性。两项研究均揭示系统性亲人类归属偏差:人类表现出+13.7个百分点(Cohen's h = 0.28,95% CI: 0.21-0.34)的偏差,而AI模型则为+34.3个百分点(h = 0.70,95% CI: 0.65-0.76),强度是人类的2.5倍(P<0.001)。研究2确认该偏差在不同AI架构间普遍存在(+25.8pp,95% CI: 24.1-27.6%),表明无论由哪个AI生成,只要被标记为“AI生成”,内容就会被系统性贬低。我们还发现,归属标签会逆转评估标准,相同特征因作者身份不同而获得相反评价。这说明AI模型在训练中吸收了人类对人工创作的偏见。本研究首次在审美判断中实现人类与人工智能评价者之间的可控对比,揭示了AI不仅复制,还放大了这种人类倾向。

原文摘要 · Abstract (English)

As AI writing tools become widespread, we need to understand how both humans and machines evaluate literary style, a domain where objective standards are elusive and judgments are inherently subjective. We conducted controlled experiments using Raymond Queneau's Exercises in Style (1947) to measure attribution bias across evaluators. Study 1 compared human participants (N=556) and AI models (N=13) evaluating literary passages from Queneau versus GPT-4-generated versions under three conditions: blind, accurately labeled, and counterfactually labeled. Study 2 tested bias generalization across a 14$\times$14 matrix of AI evaluators and creators. Both studies revealed systematic pro-human attribution bias. Humans showed +13.7 percentage point (pp) bias (Cohen's h = 0.28, 95% CI: 0.21-0.34), while AI models showed +34.3 percentage point bias (h = 0.70, 95% CI: 0.65-0.76), a 2.5-fold stronger effect (P$<$0.001). Study 2 confirmed this bias operates across AI architectures (+25.8pp, 95% CI: 24.1-27.6%), demonstrating that AI systems systematically devalue creative content when labeled as "AI-generated" regardless of which AI created it. We also find that attribution labels cause evaluators to invert assessment criteria, with identical features receiving opposing evaluations based solely on perceived authorship. This suggests AI models have absorbed human cultural biases against artificial creativity during training. Our study represents the first controlled comparison of attribution bias between human and artificial evaluators in aesthetic judgment, revealing that AI systems not only replicate but amplify this human tendency.

AI偏见文学评估人机比较

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。