用剪枝替代微调，让大模型更易优化且效果更好
Optimization over Trained (and Sparse) Neural Networks: A Surrogate within a Surrogate
- 用剪枝构建原网络的代理模型，提升优化可解性
- 不微调剪枝后反而在验证与最大化任务中表现更优
- 适合需要快速求解的优化场景，尤其大型神经网络
在约束学习中，神经网络常被用作优化模型中部分约束或目标函数的代理。但所用网络规模直接影响模型可解性。一种提升可解性的方法是先对网络进行剪枝。本文研究的是：当神经网络已给定（不可修改）时，如何求解嵌入该大网络的优化模型？我们提出对神经网络本身进行剪枝以构建更稀疏、更易处理的代理模型，期望其在原始网络上仍能获得良好解。对于网络验证和函数最大化问题，该方法确实在时间限制内得到更优解，尤其令人意外的是——跳过常规的微调步骤反而表现更好。因此，未微调的剪枝网络在某些情况下反而是更优的代理。
原文摘要 · Abstract (English)
In constraint learning, we use a neural network as a surrogate for part of the constraints or of the objective function of an optimization model. However, the tractability of the resulting model is heavily influenced by the size of the neural network used as a surrogate. One way to obtain a more tractable surrogate is by pruning the neural network first. In this work, we consider how to approach the setting in which the neural network is actually a given: how can we solve an optimization model embedding a large and predetermined neural network? We propose surrogating the neural network itself by pruning it, which leads to a sparse and more tractable optimization model, for which we hope to still obtain good solutions with respect to the original neural network. For network verification and function maximization models, that indeed leads to better solutions within a time limit, especially -- and surprisingly -- if we skip the standard retraining step known as finetuning. Hence, a pruned network with worse inference for lack of finetuning can be a better surrogate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。