根据输入复杂度动态调整稀疏性，让语言模型表示更易懂
AdaptiveK: Complexity-Driven Sparse Autoencoders for Interpretable Language Model Representations
- 按输入语义复杂度动态调节稀疏程度，提升特征表达能力
- 在10个模型上验证，重建精度与可解释性均优于固定稀疏方法
- 无需调参，适合想理解大模型内部机制的研究者
理解大语言模型（LLM）的内部表征仍是可解释性研究的核心挑战。稀疏自编码器（SAE）通过将激活分解为可解释特征提供了一种有前景的解决方案，但现有方法依赖固定的稀疏约束，无法适应输入复杂度变化。本文提出自适应K SAE（Adaptive Top K Sparse Autoencoders），一种新型框架，根据每个输入的语义复杂度动态调整稀疏水平。利用线性探测，我们证明上下文复杂度在线性层中被线性编码，并以此信号指导训练中的特征分配。在10个语言模型上的实验表明，该复杂度驱动的自适应方法在重建保真度、解释方差、余弦相似度和可解释性指标上均优于固定稀疏方法，同时免去了繁琐的超参数调优。代码已开源：https://github.com/hiyukie/adaptiveK。
原文摘要 · Abstract (English)
Understanding the internal representations of large language models (LLMs) remains a central challenge for interpretability research. Sparse autoencoders (SAEs) offer a promising solution by decomposing activations into interpretable features, but existing approaches rely on fixed sparsity constraints that fail to account for input complexity. We propose AdaptiveK SAE (Adaptive Top K Sparse Autoencoders), a novel framework that dynamically adjusts sparsity levels based on the semantic complexity of each input. Leveraging linear probes, we demonstrate that context complexity is linearly encoded in LLM representations, and we use this signal to guide feature allocation during training. Experiments across ten language models demonstrate that this complexity-driven adaptation outperforms fixed-sparsity approaches on reconstruction fidelity, explained variance, cosine similarity and interpretability metrics while eliminating the burden of extensive hyperparameter tuning. Our code is available at: https://github.com/hiyukie/adaptiveK.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。