arXiv:2602.12403cs.CV2026-02被引 1

提出MonoLoss训练目标,让神经网络特征更单一可解释。

MonoLoss: A Training Objective for Interpretable Monosemantic Representations

  • 设计线性复杂度的MonoScore计算算法,大幅提升评估效率。
  • 使用MonoLoss后,特征纯度从0.152提升至0.723,显著增强可解释性。
  • 适用于图像模型微调,带来0.6%精度提升,适合关注模型可解释性的研究者。

稀疏自编码器(SAEs)将多义神经表示分解为单义特征,使其捕捉单一可解释概念。然而,现有训练目标对这种分解的鼓励较弱,且单义性评估依赖成对比较,效率低下。本文研究最近的MonoScore指标,推导出一种单遍算法,计算结果完全一致但时间复杂度从平方级降至线性。在OpenImagesV7上,评估速度提升最高达1200倍,训练阶段提速159倍,每轮仅增加约4%开销。由此可将MonoScore作为训练信号,提出单义性损失(MonoLoss),直接奖励语义一致的激活,以学习可解释的单义表示。在CLIP、SigLIP2和预训练ViT特征上,使用BatchTopK、TopK和JumpReLU SAEs,MonoLoss显著提升多数隐层的MonoScore。同时,各类编码器与SAE组合下,类别纯度均获提升,最大增幅使基线纯度从0.152升至0.723。作为辅助正则项用于ResNet-50和CLIP-ViT-B/32微调时,可在ImageNet-1K上带来最高0.6%准确率提升,并在标准数据集上生成单义激活模式。代码已公开于https://github.com/AtlasAnalyticsLab/MonoLoss。

原文摘要 · Abstract (English)

Sparse autoencoders (SAEs) decompose polysemantic neural representations, where neurons respond to multiple unrelated concepts, into monosemantic features that capture single, interpretable concepts. However, standard training objectives only weakly encourage this decomposition, and existing monosemanticity metrics require pairwise comparisons across all dataset samples, making them inefficient during training and evaluation. We study a recent MonoScore metric and derive a single-pass algorithm that computes exactly the same quantity, but with a cost that grows linearly, rather than quadratically, with the number of dataset images. On OpenImagesV7, we achieve up to a 1200x speedup wall-clock speedup in evaluation and 159x during training, while adding only ~4% per-epoch overhead. This allows us to treat MonoScore as a training signal: we introduce the Monosemanticity Loss (MonoLoss), a plug-in objective that directly rewards semantically consistent activations for learning interpretable monosemantic representations. Across SAEs trained on CLIP, SigLIP2, and pretrained ViT features, using BatchTopK, TopK, and JumpReLU SAEs, MonoLoss increases MonoScore for most latents. MonoLoss also consistently improves class purity (the fraction of a latent's activating images belonging to its dominant class) across all encoder and SAE combinations, with the largest gain raising baseline purity from 0.152 to 0.723. Used as an auxiliary regularizer during ResNet-50 and CLIP-ViT-B/32 finetuning, MonoLoss yields up to 0.6\% accuracy gains on ImageNet-1K and monosemantic activating patterns on standard benchmark datasets. The code is publicly available at https://github.com/AtlasAnalyticsLab/MonoLoss.

可解释性自编码器单义性模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。