用机器学习检测学生作业中的AI文本，提升教育诚信。
Detecting AI-Generated Text in Educational Content: Leveraging Machine Learning and Explainable AI for Academic Integrity
- 构建1000份样本的CyberHumanAI数据集，对比真人与ChatGPT写作。
- XGBoost模型达83%准确率，短文本比长文本更难区分。
- 揭示人类写作偏实用词汇，AI文本多抽象正式表达，适合教师使用。
本研究旨在通过先进科技增强学术诚信，提供检测学生作业中AI生成内容的工具。关键贡献是构建了包含1000个样本的CyberHumanAI数据集，其中500份为真人撰写，500份由ChatGPT生成。在该数据集上评估多种机器学习（ML）与深度学习（DL）算法，比较人类与大型语言模型（LLMs，如ChatGPT）生成内容的分类效果。结果显示，传统机器学习算法表现优异，XGBoost与随机森林分别达到83%和81%准确率。结果还表明，短文本分类难度高于长文本。进一步结合可解释人工智能（XAI）分析发现，人类文本倾向使用实用语言（如use、allow），而AI文本则多含抽象正式词汇（如realm、employ）。与GPTZero对比显示，本研究提出的专注、简单且微调的模型，在纯AI、纯人类及混合类别的分类任务中准确率达约77.5%，优于GPTZero的48.5%。GPTZero在小样本和挑战性案例中常误判为混合或无法识别，而本模型在三类间表现更均衡。
原文摘要 · Abstract (English)
This study seeks to enhance academic integrity by providing tools to detect AI-generated content in student work using advanced technologies. The findings promote transparency and accountability, helping educators maintain ethical standards and supporting the responsible integration of AI in education. A key contribution of this work is the generation of the CyberHumanAI dataset, which has 1000 observations, 500 of which are written by humans and the other 500 produced by ChatGPT. We evaluate various machine learning (ML) and deep learning (DL) algorithms on the CyberHumanAI dataset comparing human-written and AI-generated content from Large Language Models (LLMs) (i.e., ChatGPT). Results demonstrate that traditional ML algorithms, specifically XGBoost and Random Forest, achieve high performance (83% and 81% accuracies respectively). Results also show that classifying shorter content seems to be more challenging than classifying longer content. Further, using Explainable Artificial Intelligence (XAI) we identify discriminative features influencing the ML model's predictions, where human-written content tends to use a practical language (e.g., use and allow). Meanwhile AI-generated text is characterized by more abstract and formal terms (e.g., realm and employ). Finally, a comparative analysis with GPTZero show that our narrowly focused, simple, and fine-tuned model can outperform generalized systems like GPTZero. The proposed model achieved approximately 77.5% accuracy compared to GPTZero's 48.5% accuracy when tasked to classify Pure AI, Pure Human, and mixed class. GPTZero showed a tendency to classify challenging and small-content cases as either mixed or unrecognized while our proposed model showed a more balanced performance across the three classes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。