改进块稀疏特征提取器，减少特征分裂问题。
A Deeper Analysis of Block-Sparse Featurizers
- 用小子空间块替代单方向，提升低维流形特征捕捉能力
- 发现原方法仍存在特征分裂和组合缺陷，影响表征质量
- 提出锦标赛式Top-K选择，显著降低特征分裂，适用于视觉任务
近期提出的块稀疏特征提取器（BSF；Fel等，2026）类似于稀疏自编码器（SAE），但其基本单元是小的子空间（方向块）而非单一方向。该方法专为定义在低维流形上的特征设计，这类特征在视觉任务中尤为常见。本文深入分析了BSF的优势与局限，发现其仍存在经典的SAE失败模式，如特征分裂和组合问题。为此，我们提出了若干架构改进，包括一种锦标赛式Top-K选择规则，能显著减少特征分裂现象，并将块范式扩展至交叉编码器（crosscoder）。
原文摘要 · Abstract (English)
The recently introduced block-sparse featurizer (BSF; Fel et al., 2026) is similar to a sparse autoencoder (SAE), but its atomic unit is a small subspace (a block of directions) rather than a single direction. It is designed for features that live on low-dimensional manifolds, which are especially frequent in vision. This work studies the BSF's strengths and weaknesses, finding how it still somewhat suffers from classic SAE failure modes, like feature splitting and composition. We propose several architectural changes to the BSF, including a Tournament Top-K selection rule that significantly reduces feature splitting, and we also extend the block paradigm to the crosscoder.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。