揭示深度模型泛化与识别的通用极限,发现语义分辨率决定性能边界。
Bound by semanticity: universal laws governing the generalization-identification tradeoff
- 基于有限语义分辨率推导出泛化与识别概率的普适帕累托前沿。
- 多输入场景下处理能力随输入数呈1/n急剧下降,泛化率存在非单调最优。
- 从简单网络到视觉-语言模型均验证该规律,适用于真实复杂系统。
智能系统需在表征结构化(支持广泛泛化)与选择性(保留输入身份)间取得平衡。本文揭示这一权衡的根本限制:对于任意表征相似度随有限语义分辨率ε衰减的模型,其正确泛化概率p_S与识别概率p_I被封闭形式表达式约束于与输入空间几何无关的普适帕累托前沿。扩展分析至噪声、异构空间及n>2个输入情形,预测多输入处理能力出现1/n的尖锐衰减,且泛化概率存在非单调最优。一个端到端训练的最小ReLU网络重现了这些规律:学习过程中自组织形成分辨率边界,经验$(p_S, p_I)$轨迹紧密跟随线性相似度衰减的理论曲线。最后,在卷积神经网络与先进视觉-语言模型中也证实该极限依然成立,表明有限分辨率相似度是深层网络与大脑共有的基本信息约束,而非玩具模型的伪象。这些结果共同构建了泛化-识别权衡的精确理论,阐明语义分辨率如何塑造深度网络与大脑的表征容量。
原文摘要 · Abstract (English)
Intelligent systems must deploy internal representations that are simultaneously structured -- to support broad generalization -- and selective -- to preserve input identity. We expose a fundamental limit on this tradeoff. For any model whose representational similarity between inputs decays with finite semantic resolution $\varepsilon$, we derive closed-form expressions that pin its probability of correct generalization $p_S$ and identification $p_I$ to a universal Pareto front independent of input space geometry. Extending the analysis to noisy, heterogeneous spaces and to $n>2$ inputs predicts a sharp $1/n$ collapse of multi-input processing capacity and a non-monotonic optimum for $p_S$. A minimal ReLU network trained end-to-end reproduces these laws: during learning a resolution boundary self-organizes and empirical $(p_S,p_I)$ trajectories closely follow theoretical curves for linearly decaying similarity. Finally, we demonstrate that the same limits persist in two markedly more complex settings -- a convolutional neural network and state-of-the-art vision-language models -- confirming that finite-resolution similarity is a fundamental emergent informational constraint, not merely a toy-model artifact. Together, these results provide an exact theory of the generalization-identification trade-off and clarify how semantic resolution shapes the representational capacity of deep networks and brains alike.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。