arXiv:2511.06786cs.LG2025-11被引 2

用图着色重新定义参数共享,实现跨层高效压缩。

Rethinking Parameter Sharing as Graph Coloring for Structured Compression

  • 将参数共享建模为图着色问题,通过结构对称性确定共享组。
  • 基于海森矩阵低曲率子空间设计准则,提升压缩后模型性能。
  • 适合追求高压缩比且保持精度的模型部署场景。

现代深度模型参数量庞大,导致推理时内存占用过高,限制实际部署。参数共享作为一种结构化压缩手段,能有效减少冗余,但现有方法仅限于相邻层间共享,缺乏对跨层共享的系统分析。将共享扩展至多层会导致配置空间指数级增长,穷举搜索计算不可行,成为参数共享的关键瓶颈。本文从群论视角重新审视参数共享,将其视为在模型参数空间引入结构对称性。共享配置可由着色函数 α: L → C 描述(L:层索引,C:共享类别),决定层间共享分组并保持结构对称性。为确定该函数,提出基于泰勒展开与海森谱的二阶几何准则。通过将扰动投影至海森矩阵的低曲率特征子空间,该准则提供一种解析规则,用于选择对性能影响最小的共享组,从而实现原理清晰且可扩展的配置流程。在多种架构与任务上,Geo-Sharing 均显著优于现有启发式共享策略,在更高压缩比下保持更小的精度损失。

原文摘要 · Abstract (English)

Modern deep models have massive parameter sizes, leading to high inference-time memory usage that limits practical deployment. Parameter sharing, a form of structured compression, effectively reduces redundancy, but existing approaches remain heuristic-restricted to adjacent layers and lacking a systematic analysis for cross-layer sharing. However, extending sharing across multiple layers leads to an exponentially expanding configuration space, making exhaustive search computationally infeasible and forming a critical bottleneck for parameter sharing. We recast parameter sharing from a group-theoretic perspective as introducing structural symmetries in the model's parameter space. A sharing configuration can be described by a coloring function $α:L\rightarrow C$ (L: layer indices and C: sharing classes), which determines inter-layer sharing groups while preserving structural symmetry. To determine the coloring function, we propose a second-order geometric criterion based on Taylor expansion and the Hessian spectrum. By projecting perturbations onto the Hessian's low-curvature eigensubspace, the criterion provides an analytic rule for selecting sharing groups that minimize performance impact, yielding a principled and scalable configuration procedure. Across diverse architectures and tasks, Geo-Sharing consistently outperforms state-of-the-art heuristic sharing strategies, achieving higher compression ratios with smaller accuracy degradation.

参数共享结构压缩图着色模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。