模型宽但内在维度低，可直接压缩到16倍小而不丢性能。
High-Dimensional Search, Low-Dimensional Solution: Decoupling Optimization from Representation
- 用随机投影分离优化与表征，不依赖数据实现压缩
- ResNet/ViT/BERT压缩16倍，性能下降仅约1%
- 支持直接训练小模型在低维流形上，适合部署优化
当前先进模型虽参数量巨大，但其内在维度（ID）较低。我们提出，这种冗余主要服务于非凸优化搜索，而非最终表征。通过数据无关的随机投影解耦解空间几何，验证了ResNet、ViT和BERT的表征可压缩至原大小的1/16，性能损失约1%。值得注意的是，这些无感知投影效果与PCA及学习型基线相当，证明解流形本身具有内在鲁棒性。该发现奠定了「子空间原生蒸馏」的基础：学生模型直接在内在流形上训练，跳过高维优化瓶颈，实现‘大模型训练，小模型部署’的愿景。
原文摘要 · Abstract (English)
State-of-the-art models rely on massive widths despite exhibiting low Intrinsic Dimension (ID). We posit that this redundancy serves the non-convex optimization search rather than the final representation. We validate this hypothesis by decoupling the solution geometry via data-independent random projections, demonstrating that ResNet, ViT, and BERT representations can be compressed by up to 16x with negligible performance degradation of around 1%. Notably, these oblivious projections achieve parity with PCA and learned baselines, confirming the solution manifold is intrinsically robust. These findings establish the foundation for Subspace-Native Distillation: a paradigm where student models target this intrinsic manifold directly, bypassing the high-dimensional optimization bottleneck to realize the vision of "Train Big, Deploy Small"
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。