用12万+虚拟人测试大模型预测社交媒体反应，发现其效果不如传统文本分类器。
LLM Agents Predict Social Media Reactions but Do Not Outperform Text Classifiers: Benchmarking Simulation Accuracy Using 120K+ Personas of 1511 Humans
- 基于零样本人格提示构建12万+虚拟用户，模拟真实人类反应行为。
- 整体预测准确率70.7%，但低于使用TF-IDF的文本分类器（MCC 0.36）。
- 揭示大模型代理可被用于社会极化模拟，但需警惕潜在操纵风险。
社交媒体平台影响数十亿人形成观点与参与公共讨论。随着人工智能代理在这些空间中的参与度日益增加，理解其行为真实性对平台治理和民主韧性至关重要。以往研究显示大语言模型驱动的代理能复现群体调查结果，但很少检验其是否能预测特定个体对特定内容的反应。本研究在来自1,511名塞尔维亚参与者、涵盖27个大语言模型的12万+唯一代理-人格组合上，基准测试了大模型代理在预测社交媒体反应（点赞、点踩、评论、分享、无反应）方面的准确性。研究1中，代理整体准确率达70.7%，不同大模型间性能差异达13个百分点。研究2采用二元强制选择（点赞/点踩）评估，使用校正随机水平的指标。代理获得0.29的马修相关系数（MCC），表明其具备超越随机的预测信号。然而，使用TF-IDF表示的传统文本监督分类器表现更优（MCC 0.36），说明其优势源于语义理解而非独特的代理推理能力。零样本人格提示代理的真实预测有效性，警示其可能被轻易部署为行为各异的代理群组以操纵社交网络，同时为模拟极化动态和制定人工智能政策提供了新工具。零样本代理的优势在于无需任务特训，便于在多元场景中大规模部署。局限在于仅单国采样。未来研究应探索多语言测试与微调方法。
原文摘要 · Abstract (English)
Social media platforms mediate how billions form opinions and engage with public discourse. As autonomous AI agents increasingly participate in these spaces, understanding their behavioral fidelity becomes critical for platform governance and democratic resilience. Previous work demonstrates that LLM-powered agents can replicate aggregate survey responses, yet few studies test whether agents can predict specific individuals' reactions to specific content. This study benchmarks LLM-based agents' accuracy in predicting human social media reactions (like, dislike, comment, share, no reaction) across 120,000+ unique agent-persona combinations derived from 1,511 Serbian participants and 27 large language models. In Study 1, agents achieved 70.7% overall accuracy, with LLM choice producing a 13 percentage-point performance spread. Study 2 employed binary forced-choice (like/dislike) evaluation with chance-corrected metrics. Agents achieved Matthews Correlation Coefficient (MCC) of 0.29, indicating genuine predictive signal beyond chance. However, conventional text-based supervised classifiers using TF-IDF representations outperformed LLM agents (MCC of 0.36), suggesting predictive gains reflect semantic access rather than uniquely agentic reasoning. The genuine predictive validity of zero-shot persona-prompted agents warns against potential manipulation through easily deploying swarms of behaviorally distinct AI agents on social media, while simultaneously offering opportunities to use such agents in simulations for predicting polarization dynamics and informing AI policy. The advantage of using zero-shot agents is that they require no task-specific training, making their large-scale deployment easy across diverse contexts. Limitations include single-country sampling. Future research should explore multilingual testing and fine-tuning approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。