动态参数化不等于动态推理,模型依赖输入分配而非真正动态计算。
Dynamic Parameterization Is Not Dynamic Inference

- 通过缓存系数并重放验证输入依赖性,揭示模型对输入条件的敏感性。
- 静态层级配置保留98.7%以上性能差距,证明动态变化非本质优势。
- 适合关注模型真实性与推理效率评估的研究者阅读。
输入依赖的控制器系数常被当作动态推理或计算节省的证据,但这一解释混淆了系数变化、冻结模型对系数分配的依赖性以及条件执行三者。本文聚焦第二点,提出通用的冻结控制器审计原则,并实现具体方法FCA:在未扰动轨迹上缓存完整系数张量,禁用控制器,再以跨输入重分配、词元打乱和独立校准集估计的静态配置重播冻结模型。由于系数提前缓存,重播中的性能变化仅反映分配依赖性,不受扰动隐藏状态反馈影响。在七个76M FeatureGate Transformer和三个504M模型上,静态分层配置分别保留98.70%和99.43%的正确率-全局均值性能差距;层身份解释87%至96%的系数方差。FeatureGate仍执行每个Transformer块,实测推理慢30.8%。在公开的MUDDPythia-1.4B检查点上,跨输入重分配和词元打乱使NLL分别增加1.9067和2.9637。这些代价表明模型强烈依赖内容条件的跨层分配。结果表明,动态参数化本身不构成动态推理,功能动态也不意味着计算节省。关于动态模型的声明应分别报告系数变化、冻结模型的功能依赖性和实际执行情况。
原文摘要 · Abstract (English)
Input-dependent controller coefficients are often treated as evidence of dynamic inference or computational savings. This interpretation conflates three properties: coefficient variation, dependence of a frozen model on how coefficients are assigned to inputs, and conditional execution. We focus on the second property and formulate a general principle of frozen-controller auditing. We provide one concrete implementation, Frozen-Controller Auditing (FCA), which caches the complete coefficient tensor along an unperturbed trajectory, disables the controller, and replays the frozen model with cross-input reassignment, token shuffling, and static profiles estimated from an independent calibration set. Because the coefficients are cached before any intervention, performance changes under replay measure assignment dependence without feedback from recomputing the controller on perturbed hidden states. Across seven independently trained 76M FeatureGate Transformers and three 504M models, static layerwise profiles retain 98.70% and 99.43% of the Correct-to-GlobalMean performance gap, respectively. Layer identity explains 87% to 96% of the coefficient variance. FeatureGate nevertheless executes every Transformer block, and its measured inference is 30.8% slower than Dense. On the public MUDDPythia-1.4B checkpoint, cross-input reassignment and token shuffling increase NLL by 1.9067 and 2.9637, respectively. These penalties show that the model depends strongly on content-conditioned cross-layer assignment. MUDDPythia also executes every Transformer block. The results show that dynamic parameterization alone does not establish dynamic inference and that functional dynamics do not establish computational savings. Claims about dynamic models should separately report coefficient variation, functional dependence of the frozen model, and actual execution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。