模型越大越难提升性能,逻辑上也存在解不了的题。
Diminishing Returns in Expanding Generative Models and Godel-Tarski-Lob Limits
- 用任务空间框架分析生成模型能力增长规律。
- 能力提升边际收益会趋近于零,且无法解决所有逻辑问题。
- 适合关注大模型极限、理论边界的研究者阅读。
现代生成模型通过扩大模型容量、训练数据和计算资源持续改进。尽管实证研究已涵盖GAN、VAE、Transformer与扩散模型等架构的缩放行为,但其能力增长的理论极限仍不清晰。本文构建了一个通用的任务空间框架:每个系统对应全局任务空间的一个子集,能力由在固定任务分布下可解任务所占概率质量衡量。在此框架中,我们证明了在温和假设下,随着系统容量增加,可解任务的边际增益必然趋于零。因此,尽管生成系统仍可继续获得能力,但新可解任务的概率质量将渐进衰减。进一步,基于算法概率思想的复杂度加权假设类,我们给出了预测任务中边际改进的定量边界。最后,通过逻辑推理任务分析,揭示罗素不完备性、塔斯基不可定义定理与洛布定理表明,足够表达力的推理系统内始终存在未解逻辑任务。这些结果共同揭示了扩展生成系统的数学本质:长期能力增长受限于边际收益递减与内在推理的根本逻辑限制。
原文摘要 · Abstract (English)
Modern generative modelling systems are increasingly improved by expanding model capacity, training data, and computational resources. While empirical studies have documented such scaling behaviour across architectures including generative adversarial networks, variational autoencoders, transformer-based models, and diffusion models, the theoretical limits of capability growth in expanding generative systems remain poorly understood. In this paper we develop a general task-space framework for analysing expanding generative reasoning systems. Each system induces a subset of a global task space representing the tasks it can successfully solve, and system capability is measured by the probability mass of this solved-task set under a fixed task distribution. Within this framework we prove a structural result showing that, under mild assumptions, the marginal improvement in solved tasks must converge to zero as system capacity increases. Thus expanding generative systems may continue to gain capability, but the probability mass of newly solvable tasks necessarily diminishes asymptotically. We further provide a prediction-theoretic refinement based on complexity-weighted hypothesis classes inspired by algorithmic probability, yielding quantitative bounds on marginal improvement in prediction settings. Finally, we examine logical reasoning tasks and show that classical results from mathematical logic -- including Rosser incompleteness, Tarski's undefinability theorem, and Löb's theorem -- imply the persistence of unresolved logical tasks within sufficiently expressive reasoning systems. Together these results provide a mathematical perspective on the asymptotic behaviour of expanding generative systems, showing that long-run capability growth is constrained both by diminishing marginal improvements in task coverage and by fundamental logical limitations on internal reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。