通过几何对齐提升多尺度表征的信息容量，解决维度冗余与谱崩溃问题。
MIC: Maximizing Informational Capacity in Adaptive Representations via Isotropic Subspace Alignment
- 用跨相关惩罚和球面均匀性约束，对齐多粒度嵌入的几何结构。
- 在高压缩场景下显著优于基线，信息容量保持能力更强。
- 适合需要高效压缩且保留语义判别力的模型设计场景。
尽管多尺度表征学习可实现弹性维度嵌入，但嵌套子空间常面临维度冗余和谱崩溃问题。为此，我们提出MIC框架，通过各向同性子空间对齐优化多粒度嵌入的几何结构。MIC采用软坍缩正则化（SCR）通过交叉相关惩罚缓解前缀与残差子空间间的冗余，同时结合谱各向同性正则化（SIR）确保低维前缀的超球面均匀性。通过自蒸馏目标统一二者，MIC生成语义密集且保持高判别力的表示。实验表明，MIC在高压缩场景下显著优于标准基线，信息容量保持能力尤为突出。
原文摘要 · Abstract (English)
Although multi-scales representation learning enables elastic-dimension embeddings, nested subspaces often suffer from dimensional redundancy and spectral collapse. To address this, we introduce MIC, a framework that optimizes the geometric landscape of multi-granular embeddings through isotropic subspace alignment. MIC employs Soft Collapse Regularization (SCR) to mitigate redundancy between prefix and residual subspaces via cross-correlation penalties, alongside Spectral Isotropy Regularization (SIR) to ensure hyper-spherical uniformity in low-dimensional prefixes. By unifying these strategies through a self-distillation objective, MIC generates semantically dense representations that maintain high discriminative power. Our experiments demonstrate that MIC significantly outperforms standard baselines, particularly in high-compression scenarios where maintaining informational capacity is most critical.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。