首次量化社交平台上AI生成文本占比,揭示其快速增长趋势。
Are We in the AI-Generated Text World Already? Quantifying and Monitoring AIGT on Social Media
- 构建SM-D与AIGTBench数据集,训练并评估检测模型
- 发现Medium和Quora上AI文本率从1.77%升至37.03%
- 揭示AI生成文本在语言、话题、互动等方面的差异特征
社交媒体正面临越来越多的AI生成文本(AIGT)。尽管其滥用可能影响公众舆论,但目前尚不清楚其实际普遍程度。本文通过收集来自Medium、Quora和Reddit的约240万条帖子,构建SM-D数据集,并结合12个大语言模型生成的AIGT数据,建立AIGTBench基准测试集,用于训练和评估检测器。基于此,识别出表现最佳的检测器OSM-Det。将其应用于SM-D数据集,从2022年1月至2024年10月追踪AIGT在各平台的演变,以AI归属率(AAR)为指标。结果显示,Medium和Quora的AAR分别从1.77%和2.06%上升至37.03%和38.95%,而Reddit仅从1.31%增至2.45%。进一步分析表明,AIGT在语言模式、主题分布、互动水平及作者粉丝数分布等方面显著区别于人工文本。
原文摘要 · Abstract (English)
Social media platforms are experiencing a growing presence of AI-Generated Texts (AIGTs). However, the misuse of AIGTs could have profound implications for public opinion, such as spreading misinformation and manipulating narratives. Despite its importance, it remains unclear how prevalent AIGTs are on social media. To address this gap, this paper aims to quantify and monitor the AIGTs on online social media platforms. We first collect a dataset (SM-D) with around 2.4M posts from 3 major social media platforms: Medium, Quora, and Reddit. Then, we construct a diverse dataset (AIGTBench) to train and evaluate AIGT detectors. AIGTBench combines popular open-source datasets and our AIGT datasets generated from social media texts by 12 LLMs, serving as a benchmark for evaluating mainstream detectors. With this setup, we identify the best-performing detector (OSM-Det). We then apply OSM-Det to SM-D to track AIGTs across social media platforms from January 2022 to October 2024, using the AI Attribution Rate (AAR) as the metric. Specifically, Medium and Quora exhibit marked increases in AAR, rising from 1.77% to 37.03% and 2.06% to 38.95%, respectively. In contrast, Reddit shows slower growth, with AAR increasing from 1.31% to 2.45% over the same period. Our further analysis indicates that AIGTs on social media differ from human-written texts across several dimensions, including linguistic patterns, topic distributions, engagement levels, and the follower distribution of authors. We envision our analysis and findings on AIGTs in social media can shed light on future research in this domain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。