通过结构化特征学习提升信息瓶颈模型的性能与可解释性
Structured IB: Improving Information Bottleneck with Structured Feature Learning
- 引入辅助编码器提取缺失关键特征，增强表征能力
- 在更小网络下仍保持更高预测准确率和任务相关信息保留
- 适合追求模型效率与可解释性的研究人员
信息瓶颈（IB）原则已成为提升深度神经网络泛化性、鲁棒性和可解释性的有前景方法，在图像分割、文档聚类和语义通信中表现良好。其中基于拉格朗日乘子的IB拉格朗日方法被广泛采用。尽管存在多种基于变分界和神经估计器的优化方案，但其性能高度依赖设计质量，易受人为误差影响。为此，本文提出结构化信息瓶颈（Structured IB），用于探索潜在的结构化特征。通过引入辅助编码器提取缺失的有用特征，生成更具信息量的表示。实验表明，相较于原始的IB拉格朗日方法，该框架在减少网络规模的情况下仍实现更高的预测精度和更强的任务相关特征保留能力。
原文摘要 · Abstract (English)
The Information Bottleneck (IB) principle has emerged as a promising approach for enhancing the generalization, robustness, and interpretability of deep neural networks, demonstrating efficacy across image segmentation, document clustering, and semantic communication. Among IB implementations, the IB Lagrangian method, employing Lagrangian multipliers, is widely adopted. While numerous methods for the optimizations of IB Lagrangian based on variational bounds and neural estimators are feasible, their performance is highly dependent on the quality of their design, which is inherently prone to errors. To address this limitation, we introduce Structured IB, a framework for investigating potential structured features. By incorporating auxiliary encoders to extract missing informative features, we generate more informative representations. Our experiments demonstrate superior prediction accuracy and task-relevant information preservation compared to the original IB Lagrangian method, even with reduced network size.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。