传统机器学习模型与现代NLP检测器在真假文本识别上效果相当,且可解释性更强。
HULLMI: Human vs LLM identification with explainability
- 用朴素贝叶斯、随机森林等传统模型实现与现代NLP检测器相当的识别效果
- 在多个真实数据集上验证了模型性能稳定,准确率可达90%以上
- 结合LIME技术揭示判断依据,提升检测结果可信度,适合教育、医疗等场景
随着大语言模型生成内容日益接近人类水平,学术界与产业界愈发关注如何区分文本来源为‘人类’或‘AI’。现有方法多依赖T5-Sentinel、RoBERTa-Sentinel等现代NLP检测器,却忽视其可解释性问题。本研究通过在多种数据集(包括人工标注语料和真实样本)上进行严谨测试,发现朴素贝叶斯、MLP、随机森林、XGBoost等传统机器学习模型表现不逊于现代NLP检测器。进一步采用可解释AI技术LIME,揭示各模型预测中关键输入特征,深入理解判别机制。研究支持构建兼具高精度与可解释性的生产级检测工具,融合传统与现代模型优势,尤其适用于教育、医疗、媒体等对可靠性要求高的领域。
原文摘要 · Abstract (English)
As LLMs become increasingly proficient at producing human-like responses, there has been a rise of academic and industrial pursuits dedicated to flagging a given piece of text as "human" or "AI". Most of these pursuits involve modern NLP detectors like T5-Sentinel and RoBERTa-Sentinel, without paying too much attention to issues of interpretability and explainability of these models. In our study, we provide a comprehensive analysis that shows that traditional ML models (Naive-Bayes,MLP, Random Forests, XGBoost) perform as well as modern NLP detectors, in human vs AI text detection. We achieve this by implementing a robust testing procedure on diverse datasets, including curated corpora and real-world samples. Subsequently, by employing the explainable AI technique LIME, we uncover parts of the input that contribute most to the prediction of each model, providing insights into the detection process. Our study contributes to the growing need for developing production-level LLM detection tools, which can leverage a wide range of traditional as well as modern NLP detectors we propose. Finally, the LIME techniques we demonstrate also have the potential to equip these detection tools with interpretability analysis features, making them more reliable and trustworthy in various domains like education, healthcare, and media.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。