从信息论角度分析回归任务的表征学习，揭示表征能力的极限。
Information Theoretic Perspective on Representation Learning
- 基于输入源熵定义表征速率，刻画信息保留上限。
- 在扰动环境下推导出可实现的表征容量与率失真关系。
- 统一框架下揭示表征学习的理论边界，适合理论研究者。
本文提出一种信息论框架，用于分析回归任务中的最后一层嵌入表征。通过定义表征速率，揭示了输入输出信息能够被可靠表示的上限，该上限由输入源熵决定。进一步在扰动设置下定义表征容量，并推导压缩输出下的表征率失真关系。文中推导了可实现容量、可实现表征速率及其对偶下界。最终将各项结果整合进统一框架中。
原文摘要 · Abstract (English)
An information-theoretic framework is introduced to analyze last-layer embedding, focusing on learned representations for regression tasks. We define representation-rate and derive limits on the reliability with which input-output information can be represented as is inherently determined by the input-source entropy. We further define representation capacity in a perturbed setting, and representation rate-distortion for a compressed output. We derive the achievable capacity, the achievable representation-rate, and their converse. Finally, we combine the results in a unified setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。