softmax在规模扩大时难以保持精准判断，论文揭示其根本局限并提出改进方法。
Softmax is not Enough (for Sharp Size Generalisation)
- 发现softmax在输入规模增大时无法维持精准决策能力
- 理论证明：规模扩展导致计算电路分散，无法稳定输出最大值
- 提出自适应温度调节法，提升推理阶段的判断锐度
推理系统的关键能力是针对输入数据做出精确决策。当前AI系统中，softmax函数因其可微查询-键值匹配能力，成为实现精确行为的核心机制。人们普遍认为，依赖softmax的网络通过稳定执行特定计算的“电路”获得预测能力。然而，这些电路若要稳健，必须对任意有效输入具备良好泛化性。本文推翻这一认知：即使是在寻找最大键值这样简单的任务中，随着测试时项目数量增加，任何学习到的电路都必然趋于分散。我们指出，这是由softmax函数在问题规模扩大时难以稳健逼近精确函数所导致的根本限制，并从理论上证明该现象；同时提出一种自适应温度策略，作为提升推理阶段软决策锐度的临时解决方案。
原文摘要 · Abstract (English)
A key property of reasoning systems is the ability to make sharp decisions on their input data. For contemporary AI systems, a key carrier of sharp behaviour is the softmax function, with its capability to perform differentiable query-key lookups. It is a common belief that the predictive power of networks leveraging softmax arises from "circuits" which sharply perform certain kinds of computations consistently across many diverse inputs. However, for these circuits to be robust, they would need to generalise well to arbitrary valid inputs. In this paper, we dispel this myth: even for tasks as simple as finding the maximum key, any learned circuitry must disperse as the number of items grows at test time. We attribute this to a fundamental limitation of the softmax function to robustly approximate sharp functions with increasing problem size, prove this phenomenon theoretically, and propose adaptive temperature as an ad-hoc technique for improving the sharpness of softmax at inference time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。