arXiv:2510.01105cs.LG2025-10被引 1

发现神经回归坍缩会压缩特征维度,导致泛化变差。

Geometric Analysis of Neural Regression Collapse via Intrinsic Dimension

  • 用内在维度分析回归模型的几何特性
  • 坍缩模型特征维数低于目标维数,导致性能下降
  • 提出过压缩与欠压缩两种场景,指导调参策略

神经多变量回归广泛应用于控制、机器人和金融等领域,但其学习表征的几何性质仍不清楚。尽管神经坍缩在分类任务中能提升泛化能力,我们发现回归任务中的类似坍缩反而会降低性能。通过内在维度(ID_H)分析最后层特征与回归目标(ID_Y)的关系,在控制任务和合成数据集上发现:坍缩模型满足 ID_H < ID_Y,造成过度压缩,泛化能力差;而非坍缩模型通常保持 ID_H > ID_Y。非坍缩模型的性能受数据量和噪声水平影响,据此识别出过压缩与欠压缩两种状态,决定何时应扩展或缩减特征维度。研究为神经回归坍缩提供了新的几何理解,并提出改善泛化的实用方法。

原文摘要 · Abstract (English)

Neural multivariate regression underpins a wide range of domains, including control, robotics, and finance, yet the geometry of its learned representations remains poorly characterized. While neural collapse has been shown to benefit generalization in classification, we find that analogous collapse in regression consistently degrades performance. To explain this contrast, we analyze regression models through the lens of intrinsic dimension. Across control tasks and synthetic datasets, we estimate the intrinsic dimension of last-layer features (ID_H) and compare it with that of the regression targets (ID_Y). Collapsed models exhibit ID_H < ID_Y, leading to over-compression and poor generalization, whereas non-collapsed models typically maintain ID_H > ID_Y. For the non-collapsed models, performance with respect to ID_H depends on the data quantity and noise levels. From these observations, we identify two regimes (over-compressed and under-compressed) that determine when expanding or reducing feature dimensionality improves performance. Our results provide new geometric insights into neural regression collapse and suggest practical strategies for enhancing generalization.

神经回归内在维度泛化能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。