arXiv:2603.11049cs.LG2026-03

比较两种字符串异常检测算法,发现结构差异大的异常更易被正则表达式法识别。

Comparison of Outlier Detection Algorithms on String Data

  • 用编辑距离衡量字符串密度,改进局部离群因子算法适配字符串数据
  • 正则表达式法在预期数据结构明显时,对异常值识别效果更优
  • 适用于系统日志清洗等需要识别非数值异常的场景

异常检测是机器学习中的重要问题,但针对字符串数据的研究较少,多数工作聚焦于数值型数据。本论文比较了两种字符串异常检测算法。首先,提出一种基于勒文斯坦距离的局部离群因子变体,通过引入分层字符类别加权机制,提升对特定字符串数据集的适应性。其次,提出一种基于分层左正则表达式学习器的新算法,从正常数据中推断出正则表达式以刻画预期模式。实验表明,两种方法均能有效识别字符串异常。当正常数据具有显著结构特征且与异常值差异明显时,正则表达式法表现更佳;而当异常值与正常数据的编辑距离明显偏离时,局部离群因子算法更具优势。

原文摘要 · Abstract (English)

Outlier detection is a well-researched and crucial problem in machine learning. However, there is little research on string data outlier detection, as most literature focuses on outlier detection of numerical data. A robust string data outlier detection algorithm could assist with data cleaning or anomaly detection in system log files. In this thesis, we compare two string outlier detection algorithms. Firstly, we introduce a variant of the well-known local outlier factor algorithm, which we tailor to detect outliers on string data using the Levenshtein measure to calculate the density of the dataset. We present a differently weighted Levenshtein measure, which considers hierarchical character classes and can be used to tune the algorithm to a specific string dataset. Secondly, we introduce a new kind of outlier detection algorithm based on the hierarchical left regular expression learner, which infers a regular expression for the expected data. Using various datasets and parameters, we experimentally show that both algorithms can conceptually find outliers in string data. We show that the regular expression-based algorithm is especially good at finding outliers if the expected values have a distinct structure that is sufficiently different from the structure of the outliers. In contrast, the local outlier factor algorithms are best at finding outliers if their edit distance to the expected data is sufficiently distinct from the edit distance between the expected data.

异常检测字符串数据正则表达式勒文斯坦距离

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。