揭示异常检测中内点记忆效应的原理,指导提升模型性能
What Drives the Inlier-Memorization Effect? A Theory of Outlier Detection via Early Training Dynamics
- 基于自编码器分析早期训练动态,解释为何内点比异常点更早被记住
- 证明在特定训练阶段,模型能有效记忆内点而忽略异常点
- 提出数据预处理与初始化方案,显著提升异常检测效果
异常检测旨在通过学习正常数据(内点)的内在结构来识别异常实例,尤其在完全无监督设置下挑战巨大。近期方法利用内点记忆(IM)效应——深度模型比异常点更早记住内点模式——作为关键区分信号。然而,该现象的理论基础仍不明确。本文以简单自编码器为研究对象,在温和假设下证明:在早期训练阶段,模型可成功记忆内点而无法记忆异常点。我们不仅刻画了IM效应的出现、强度与持续性,还分析其如何受数据分布与参数初始化影响。基于这些发现,我们提出简洁有效的改进策略,包括数据预处理和参数初始化方法,在ADBench数据集上达到当前最优性能。研究成果为IM效应提供了理论支撑,并指明优化方向。
原文摘要 · Abstract (English)
Outlier detection (OD) aims to identify anomalous instances by learning the underlying structure of normal data (inliers), and is particularly challenging in fully unsupervised settings where no information about anomalies is available during training. Recent advances have leveraged the inlier-memorization (IM) effect, a phenomenon in which deep models memorize inlier patterns earlier than those of outliers, as a powerful signal for distinguishing outliers. However, despite its empirical success, the theoretical understanding of the IM effect remains limited. In this work, we present a theoretical study of the IM effect. Focusing on a simple autoencoder, we show that, under mild assumptions, the model can successfully memorize inliers while failing to memorize outliers during certain stages of early training. In particular, we characterize not only the emergence of the IM effect, but also its strength and persistence, and analyze how these properties depend on the data distribution and parameter initialization. In addition, building on these insights, we derive simple yet practical guidelines for enhancing the IM effect, including data preprocessing and parameter initialization schemes, achieving state-of-the-art performance on the ADBench datasets. Our findings provide a theoretical foundation for the IM effect and offer actionable directions for improving IM-based outlier detection methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。