用写作特征分析技术量化学生与AI协作程度,助力学术诚信透明化
Human-AI Collaboration or Academic Misconduct? Measuring AI Use in Student Writing Through Stylometric Evidence
- 构建学生写作特征画像,通过风格差异识别AI辅助痕迹
- 在1889份文档上实现对人机协作的逐词逐句检测,准确率显著提升
- 适合教育者用于学术诚信评估,推动可解释的AI使用监管
随着人机协作在教育场景中日益普遍,如何理解和测量其程度与性质成为挑战。本研究将作者身份验证(AV)技术应用于量化学术写作中的AI协助,旨在促进透明度、可解释性与学生发展,而非惩罚。基于前期工作,研究分为三个阶段:数据集选择与扩展、AV方法开发、系统评估。使用三个数据集——包括公开数据集PAN-14及墨尔本大学多课程学生的两个数据集——共扩充至1,889份文档,涵盖506名学生的540个作者识别问题,并加入大模型生成文本。提出改进的特征向量差分AV方法,构建稳健的学术写作个体特征档案,能捕捉有意义的写作风格特征。该方法在多种场景下评估,包括区分学生原创与模型生成文本,以及测试模型模仿学生风格的抗性。结果表明,改进后的分类器能有效识别风格差异,在词与句级别量化人机协作,为教育者提供透明可解释的工具,支持学术诚信调查。本研究推动了AV技术发展,为人工智能时代下的学术写作动态提供了可操作洞察。
原文摘要 · Abstract (English)
As human-AI collaboration becomes increasingly prevalent in educational contexts, understanding and measuring the extent and nature of such interactions pose significant challenges. This research investigates the use of authorship verification (AV) techniques not as a punitive measure, but as a means to quantify AI assistance in academic writing, with a focus on promoting transparency, interpretability, and student development. Building on prior work, we structured our investigation into three stages: dataset selection and expansion, AV method development, and systematic evaluation. Using three datasets - including a public dataset (PAN-14) and two from University of Melbourne students from various courses - we expanded the data to include LLM-generated texts, totalling 1,889 documents and 540 authorship problems from 506 students. We developed an adapted Feature Vector Difference AV methodology to construct robust academic writing profiles for students, designed to capture meaningful, individual characteristics of their writing. The method's effectiveness was evaluated across multiple scenarios, including distinguishing between student-authored and LLM-generated texts and testing resilience against LLMs' attempts to mimic student writing styles. Results demonstrate the enhanced AV classifier's ability to identify stylometric discrepancies and measure human-AI collaboration at word and sentence levels while providing educators with a transparent tool to support academic integrity investigations. This work advances AV technology, offering actionable insights into the dynamics of academic writing in an AI-driven era.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。