用有损压缩视角解析机器学习泛化，揭示过拟合与偏差本质
Supervised Learning as Lossy Compression: Characterizing Generalization and Sample Complexity via Finite Blocklength Analysis
- 将训练数据采样视为编码，模型构建视为解码，建立信息论框架
- 推导出样本复杂度与泛化误差的下界，明确区分过拟合与归纳偏置失配
- 统一了信息论界与稳定性理论的度量，适合研究泛化机制的学者
本文提出一种新的信息论视角来理解机器学习中的泛化问题，将学习过程置于有损压缩框架中，并应用有限块长分析。在该方法中，训练数据的采样形式上对应编码过程，模型构建则对应解码过程。通过有限块长分析，我们为固定随机学习算法及其最优采样策略推导出样本复杂度和泛化误差的下界。这些界显式刻画了学习算法的过拟合程度以及其归纳偏置与任务之间的不匹配程度，二者可分离，相较现有框架具有显著优势。此外，我们对过拟合项进行分解，揭示其与已有信息论界及稳定性理论度量的理论关联，将这些视角统一于新提出的框架之下。
原文摘要 · Abstract (English)
This paper presents a novel information-theoretic perspective on generalization in machine learning by framing the learning problem within the context of lossy compression and applying finite blocklength analysis. In our approach, the sampling of training data formally corresponds to an encoding process, and the model construction to a decoding process. By leveraging finite blocklength analysis, we derive lower bounds on sample complexity and generalization error for a fixed randomized learning algorithm and its associated optimal sampling strategy. Our bounds explicitly characterize the degree of overfitting of the learning algorithm and the mismatch between its inductive bias and the task as distinct terms. This separation provides a significant advantage over existing frameworks. Additionally, we decompose the overfitting term to show its theoretical connection to existing metrics found in information-theoretic bounds and stability theory, unifying these perspectives under our proposed framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。