首次实现文本异常的细粒度定位，精准找出问题字词。
Towards Token-Level Text Anomaly Detection
- 提出跨层级统一框架，同时检测文档与字词级异常
- 在3个数据集上优于6个基线模型，定位更精确
- 公开标注数据集与代码，推动细粒度异常检测研究
尽管网页应用中的文本异常检测（如垃圾信息过滤、假新闻识别）已取得显著进展，现有方法仍局限于文档级分析，无法定位具体异常片段。本文提出词元级异常检测新范式，实现文本内部异常的细粒度定位。我们形式化定义了文档与词元两级异常，并设计统一检测框架支持多层级分析。为推动该方向研究，我们构建并标注了三个基准数据集，涵盖垃圾信息、评论和语法错误，均带有词元级标签。实验表明，所提框架性能优于6个基线方法，为文本异常的精准定位开辟新路径。所有代码与数据已开源：https://github.com/charles-cao/TokenCore。
原文摘要 · Abstract (English)
Despite significant progress in text anomaly detection for web applications such as spam filtering and fake news detection, existing methods are fundamentally limited to document-level analysis, unable to identify which specific parts of a text are anomalous. We introduce token-level anomaly detection, a novel paradigm that enables fine-grained localization of anomalies within text. We formally define text anomalies at both document and token-levels, and propose a unified detection framework that operates across multiple levels. To facilitate research in this direction, we collect and annotate three benchmark datasets spanning spam, reviews and grammar errors with token-level labels. Experimental results demonstrate that our framework get better performance than other 6 baselines, opening new possibilities for precise anomaly localization in text. All the codes and data are publicly available on https://github.com/charles-cao/TokenCore.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。