arXiv:2501.05398cs.LGcs.AI2025-01被引 47

用语义空间解析神经网络内部知识,实现组件级可解释与验证。

Mechanistic understanding and validation of large AI models with SemanticLens

  • 将神经元知识映射到CLIP的语义多模态空间,实现自动分析
  • 可定位特定概念神经元、自动标注功能并检测虚假关联
  • 适合需要可解释性与可信验证的AI系统开发与审计

与飞机等人工系统不同,人工智能模型内部机制仍不透明,影响可验证性与可信度。本文提出通用解释方法SemanticLens,将神经网络中组件(如单个神经元)编码的隐含知识映射至基础模型(如CLIP)的语义结构化多模态空间。在此空间中,可实现:(i) 文本搜索定位编码特定概念的神经元;(ii) 系统化分析与比较模型表征;(iii) 自动标注神经元并解释其功能;(iv) 对决策进行合规性审计。该方法完全可扩展且无需人工干预,在调试、知识总结、推理对齐(如皮肤癌分类中的ABCDE规则遵循)、检测虚假相关性及其训练数据方面表现有效。通过实现组件级理解与验证,该方法有助于弥合AI模型与传统工程系统间的‘信任鸿沟’。代码与演示已公开于https://github.com/jim-berend/semanticlens和https://semanticlens.hhi-research-insights.eu。

原文摘要 · Abstract (English)

Unlike human-engineered systems such as aeroplanes, where each component's role and dependencies are well understood, the inner workings of AI models remain largely opaque, hindering verifiability and undermining trust. This paper introduces SemanticLens, a universal explanation method for neural networks that maps hidden knowledge encoded by components (e.g., individual neurons) into the semantically structured, multimodal space of a foundation model such as CLIP. In this space, unique operations become possible, including (i) textual search to identify neurons encoding specific concepts, (ii) systematic analysis and comparison of model representations, (iii) automated labelling of neurons and explanation of their functional roles, and (iv) audits to validate decision-making against requirements. Fully scalable and operating without human input, SemanticLens is shown to be effective for debugging and validation, summarizing model knowledge, aligning reasoning with expectations (e.g., adherence to the ABCDE-rule in melanoma classification), and detecting components tied to spurious correlations and their associated training data. By enabling component-level understanding and validation, the proposed approach helps bridge the "trust gap" between AI models and traditional engineered systems. We provide code for SemanticLens on https://github.com/jim-berend/semanticlens and a demo on https://semanticlens.hhi-research-insights.eu.

可解释性神经网络模型验证语义空间

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。