arXiv:2501.19032cs.LG2025-01AAAI被引 3

提出新方法自动发现模型易错数据片段,无需额外标签信息。

Error Slice Discovery via Manifold Compactness

  • 基于数据几何结构设计无监督一致性度量
  • 在多个数据集上验证了算法识别错误片段的有效性
  • 适合希望分析模型弱点的研究者与工程师

尽管深度学习模型在诸多领域表现优异,仍会在某些数据子集上出错,即存在错误片段。准确识别语义一致且易于理解的错误片段(即错误片段发现)至关重要。然而,现有方法依赖预定义标签(如属性或子类)来评估片段一致性,其有效性严重依赖于元数据的质量与数量,可能导致部分模式被忽略。同时,由于缺乏显式的统一一致性度量,现有算法无法直接将一致性作为优化目标,可能影响性能。本文提出“流形紧凑性”(Manifold Compactness),一种不依赖额外信息的一致性度量,融合数据几何特性设计,并通过典型数据集实验证实其合理性。进一步提出基于流形紧凑性的错误片段发现(MCSD)算法,可直接将风险与一致性纳入优化目标,适用于多种任务模型。大量基准实验及案例研究证明了该方法的优越性。

原文摘要 · Abstract (English)

Despite the great performance of deep learning models in many areas, they still make mistakes and underperform on certain subsets of data, i.e. error slices. Given a trained model, it is important to identify its semantically coherent error slices that are easy to interpret, which is referred to as the error slice discovery problem. However, there is no proper metric of slice coherence without relying on extra information like predefined slice labels. Current evaluation of slice coherence requires access to predefined slices formulated by metadata like attributes or subclasses. Its validity heavily relies on the quality and abundance of metadata, where some possible patterns could be ignored. Besides, current algorithms cannot directly incorporate the constraint of coherence into their optimization objective due to absence of an explicit coherence metric, which could potentially hinder their effectiveness. In this paper, we propose manifold compactness, a coherence metric without reliance on extra information by incorporating the data geometry property into its design, and experiments on typical datasets empirically validate the rationality of the metric. Then we develop Manifold Compactness based error Slice Discovery (MCSD), a novel algorithm that directly treats risk and coherence as the optimization objective, and is flexible to be applied to models of various tasks. Extensive experiments on the benchmark and case studies on other typical datasets demonstrate the superiority of MCSD.

错误分析流形学习模型诊断

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。