对比多种AI文本检测方法,发现上下文建模优于词法特征。
AI Generated Text Detection
- 用主题划分数据集防泄露,提升模型泛化能力
- DistilBERT达88.11%准确率,ROC-AUC高达0.96
- 适合关注学术诚信与AI检测的教育研究者
大型语言模型的快速发展导致生成文本增多,学生滥用生成内容违反学术诚信。本文评估了传统机器学习与基于Transformer的检测方法,使用HC3和DAIGT v2两个数据集构建统一基准,并采用基于主题的数据划分以防止信息泄露,确保跨未见领域的鲁棒性。实验表明,TF-IDF逻辑回归达到82.87%基准准确率;深度学习模型表现更优,BiLSTM准确率达88.86%,DistilBERT准确率88.11%,并取得最高ROC-AUC 0.96,显示其最强综合性能。结果表明,上下文语义建模显著优于词法特征,强调通过合理评估协议缓解主题记忆的重要性。当前局限在于数据集多样性不足与计算资源限制。未来工作将扩展数据多样性,采用LoRA等参数高效微调方法,探索小型或蒸馏模型,优化批处理策略与硬件感知计算。
原文摘要 · Abstract (English)
The rapid development of large language models has led to an increase in AI-generated text, with students increasingly using LLM-generated content as their own work, which violates academic integrity. This paper presents an evaluation of AI text detection methods, including both traditional machine learning models and transformer-based architectures. We utilize two datasets, HC3 and DAIGT v2, to build a unified benchmark and apply a topic-based data split to prevent information leakage. This approach ensures robust generalization across unseen domains. Our experiments show that TF-IDF logistic regression achieves a reasonable baseline accuracy of 82.87%. However, deep learning models outperform it. The BiLSTM classifier achieves an accuracy of 88.86%, while DistilBERT achieves a similar accuracy of 88.11% with the highest ROC-AUC score of 0.96, demonstrating the strongest overall performance. The results indicate that contextual semantic modeling is significantly superior to lexical features and highlight the importance of mitigating topic memorization through appropriate evaluation protocols. The limitations of this work are primarily related to dataset diversity and computational constraints. In future work, we plan to expand dataset diversity and utilize parameter-efficient fine-tuning methods such as LoRA. We also plan to explore smaller or distilled models and employ more efficient batching strategies and hardware-aware optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。