提出新方法量化人机协作写作中的参与度，突破传统二元检测局限。
Measuring Human Involvement in AI-Generated Text: A Case Study on Academic Writing
- 用BERTScore衡量人类参与程度，结合多任务RoBERTa回归器建模。
- 在连续参与度数据集上实现F1 0.9423，均方误差仅0.004。
- 适用于不同生成模型，适合教育、学术审查等场景使用。
大型语言模型如ChatGPT和Claude的快速发展极大推动了内容创作，但也引发社会担忧。一项调查显示近30%大学生使用生成式AI撰写论文与报告。现有检测方法多将文本归属视为二分类任务，忽视了人机协作中人类参与程度的连续变化，导致检测失效。本文称此为‘参与度检测混淆’。我们提出以BERTScore作为人类参与度度量指标，并构建基于RoBERTa的多任务回归模型，在模拟学术场景下构建了反映不同参与水平的连续数据集。所有现有检测器在该数据集上均无法识别参与程度，而我们的方法成功实现高精度预测（F1分数0.9423，均方误差0.004），并在不同生成模型间展现出一定泛化能力。代码已开源。
原文摘要 · Abstract (English)
Content creation has dramatically progressed with the rapid advancement of large language models like ChatGPT and Claude. While this progress has greatly enhanced various aspects of life and work, it has also negatively affected certain areas of society. A recent survey revealed that nearly 30% of college students use generative AI to help write academic papers and reports. Most countermeasures treat the detection of AI-generated text as a binary classification task and thus lack robustness. This approach overlooks human involvement in the generation of content even though human-machine collaboration is becoming mainstream. Besides generating entire texts, people may use machines to complete or revise texts. Such human involvement varies case by case, which makes binary classification a less than satisfactory approach. We refer to this situation as participation detection obfuscation. We propose using BERTScore as a metric to measure human involvement in the generation process and a multi-task RoBERTa-based regressor trained on a token classification task to address this problem. To evaluate the effectiveness of this approach, we simulated academic-based scenarios and created a continuous dataset reflecting various levels of human involvement. All of the existing detectors we examined failed to detect the level of human involvement on this dataset. Our method, however, succeeded (F1 score of 0.9423 and a regressor mean squared error of 0.004). Moreover, it demonstrated some generalizability across generative models. Our code is available at https://github.com/gyc-nii/CAS-CS-and-dual-head-detector
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。