用大模型当文学批评家，发现它们各有独特审美偏好。
Subjective Evaluation Profile Analysis of Science Fiction Short Stories and its Critical-Theoretical Significance
- 让六款顶尖大模型评价十篇科幻短篇，分七次进行
- 评估一致性差异达4.5倍，模型间词汇偏好截然不同
- 揭示大模型受训练影响，像人类评论流派有固定风格
本研究将大型语言模型（LLMs）视为“主观文学批评者”，探究文学评价中的审美偏好与评估模式。十篇日文科幻短篇被译为英文，由六款先进大模型在七个独立会话中进行评估。主成分分析与聚类技术显示，评估一致性存在显著差异（α值从1.00到0.35），并识别出五种不同的评估模式。此外，不同故事间的评估方差相差最高达4.5倍，TF-IDF分析证实各模型具有独特的评价词汇体系。通过使用原创科幻语料库及七次日内重复实验协议，有效控制外部偏见，从而揭示了受强化学习人类反馈（RLHF）影响的隐含价值体系对文学判断的作用。研究结果表明，大模型并非中立评判标准，而是具备类似人类批评学派的个体化评价特征。
原文摘要 · Abstract (English)
This study positions large language models (LLMs) as "subjective literary critics" to explore aesthetic preferences and evaluation patterns in literary assessment. Ten Japanese science fiction short stories were translated into English and evaluated by six state-of-the-art LLMs across seven independent sessions. Principal component analysis and clustering techniques revealed significant variations in evaluation consistency (α ranging from 1.00 to 0.35) and five distinct evaluation patterns. Additionally, evaluation variance across stories differed by up to 4.5-fold, with TF-IDF analysis confirming distinctive evaluation vocabularies for each model. Our seven-session within-day protocol using an original Science Fiction corpus strategically minimizes external biases, allowing us to observe implicit value systems shaped by RLHF and their influence on literary judgment. These findings suggest that LLMs may possess individual evaluation characteristics similar to human critical schools, rather than functioning as neutral benchmarkers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。