arXiv:2509.03733cs.LGcs.AI2025-09被引 6

通过可微熵正则化降低模型复杂度,实现更快更鲁棒的神经网络优化。

Differentiable Entropy Regularization: A Complexity-Aware Approach for Neural Optimization

  • 提出首个可微的范围分割熵近似,直接约束算法复杂度
  • 在ImageNet上实现80%稀疏下80.1%准确率,推理速度提升2.07倍
  • 适用于视觉与几何任务,提升模型鲁棒性,适合追求高效可靠的部署

我们提出首个可微的范围分割熵近似,这是一种来自计算几何的复杂度度量,能直接约束算法运行时间。该方法作为互补正则项,不依赖架构修改,与现有优化结合可获得正交效率增益。理论证明在凸包和三角剖分任务中实现4–5倍可证提速,误差低于0.2%。在ImageNet-1K上,ViT-Base模型在80%稀疏度下达到80.1%准确率,独立提速1.60倍;与FlashAttention结合后达2.07倍,优于FlashAttention单独使用的1.63倍。在大语言模型(LLaMA-2 7B、Mistral-7B、Phi-2)上,70–75%稀疏度下实现1.48–1.60倍推理加速,质量损失极小(ROUGE-L下降0.3–0.4,困惑度上升0.9)。不同于以往针对输出分布的正则化,本方法直接最小化表示复杂度,生成语义结构化的稀疏模式,显著提升鲁棒性(IoU 0.73 vs 0.41,CIFAR-100-C mCE 48.7 vs 55.4)。优势在几何与视觉变换器中最强,大语言模型也有可观收益,表明复杂度正则化是实现效率与鲁棒性协同优化的可靠路径。

原文摘要 · Abstract (English)

We introduce the first differentiable approximation of range-partition entropy, a complexity measure from computational geometry that directly bounds algorithmic runtime. Unlike architectural modifications, our method is a complementary regularizer that provides orthogonal efficiency gains when combined with existing optimizations. We establish theoretical guarantees in computational geometry, achieving 4--5$\times$ provable speedups on convex hull and triangulation with $<$0.2\% error. On ImageNet-1K with ViT-Base, entropy regularization achieves 80.1\% top-1 accuracy at 80\% sparsity (1.60$\times$ standalone speedup), and when combined with FlashAttention yields 2.07$\times$ speedup versus 1.63$\times$ for FlashAttention alone. On large language models (LLaMA-2 7B, Mistral-7B, Phi-2), we achieve 1.48--1.60$\times$ inference speedups at 70--75\% sparsity with minimal quality degradation (ROUGE-L drops of 0.3--0.4 points, perplexity increase of 0.9). Unlike prior regularization methods that target output distributions, we directly minimize representation complexity, yielding both efficiency gains and improved robustness through semantically structured sparsity patterns (IoU 0.73 vs 0.41 for magnitude pruning, CIFAR-100-C mCE 48.7 vs 55.4). Benefits are strongest for geometry and vision transformers, with more modest but measurable gains on LLMs, demonstrating that complexity regularization offers a principled pathway to joint efficiency-robustness optimization.

神经优化熵正则化稀疏化高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。