提出抗标签噪声的信息瓶颈方法,提升模型在错误标签下的学习鲁棒性。
Is the Information Bottleneck Robust Enough? Towards Label-Noise Resistant Information Bottleneck Learning
- 引入最小充分清洁准则,分离干净标签与噪声信息。
- 三阶段训练框架显著降低噪声干扰,提升模型泛化能力。
- 理论证明可使表示对输入噪声不变,适合真实噪声数据场景。
信息瓶颈(IB)原则通过保留与标签相关的信息并压缩无关信息,促进有效表征学习。然而,其对准确标签的高度依赖使其在存在标签噪声的真实场景中极易失效,导致性能下降和过拟合。为此,我们提出LaT-IB——一种新型抗标签噪声信息瓶颈方法,引入‘最小充分清洁’(MSC)准则,作为互信息正则项,以保留任务相关信号并丢弃噪声。LaT-IB采用噪声感知的潜在因子解耦机制,将隐变量分解为与干净标签空间和噪声空间对齐的成分。理论上,我们推导了预测、压缩与解耦目标的互信息上界,并证明优化该目标可促使表示对输入噪声不变,且分离干净与噪声标签信息。此外,设计了三阶段训练框架:预热、知识注入与鲁棒训练,逐步引导模型获得抗噪表征。大量实验表明,LaT-IB在标签噪声下表现出更优的鲁棒性与效率,显著增强在真实噪声场景中的适用性。
原文摘要 · Abstract (English)
The Information Bottleneck (IB) principle facilitates effective representation learning by preserving label-relevant information while compressing irrelevant information. However, its strong reliance on accurate labels makes it inherently vulnerable to label noise, prevalent in real-world scenarios, resulting in significant performance degradation and overfitting. To address this issue, we propose LaT-IB, a novel Label-Noise ResistanT Information Bottleneck method which introduces a "Minimal-Sufficient-Clean" (MSC) criterion. Instantiated as a mutual information regularizer to retain task-relevant information while discarding noise, MSC addresses standard IB's vulnerability to noisy label supervision. To achieve this, LaT-IB employs a noise-aware latent disentanglement that decomposes the latent representation into components aligned with to the clean label space and the noise space. Theoretically, we first derive mutual information bounds for each component of our objective including prediction, compression, and disentanglement, and moreover prove that optimizing it encourages representations invariant to input noise and separates clean and noisy label information. Furthermore, we design a three-phase training framework: Warmup, Knowledge Injection and Robust Training, to progressively guide the model toward noise-resistant representations. Extensive experiments demonstrate that LaT-IB achieves superior robustness and efficiency under label noise, significantly enhancing robustness and applicability in real-world scenarios with label noise.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。