arXiv:2508.16905cs.LGcs.AI2025-08

动态协调精度、二阶信息与批处理,让模型训练更快更省显存。

Tri-Accel: Curvature-Aware Precision-Adaptive and Memory-Elastic Optimization for Efficient GPU Usage

  • 根据曲率与梯度方差自动分配各层精度
  • 利用海塞矩阵稀疏性优化步长与精度,提升效率13.3%
  • 显存不足时自动调小批次,适合边缘设备部署

深度神经网络的训练正面临优化成本过高问题,尤其在显存和计算时间方面。现有加速技术如混合精度、二阶方法和批量扩展通常独立使用。本文提出统一优化框架Tri-Accel,联合自适应三种策略:(1) 基于曲率与梯度方差动态分配各层混合精度;(2) 利用海塞/费舍尔矩阵的稀疏模式指导精度与步长选择;(3) 根据显存实时调整批量大小。在CIFAR-10上使用ResNet-18和EfficientNet-B0,Tri-Accel实现最高9.9%训练时间减少和13.3%内存降低,且准确率比FP32基准提升1.1个百分点。在CIFAR-10/100上的测试显示,系统随训练过程逐渐优化资源分配。相比静态混合精度,其在标准硬件上将内存从0.35GB降至0.31GB,仍保持78.1%准确率。框架采用定制Triton内核,实现硬件感知自动优化,无需人工调参,适用于多样计算环境。

原文摘要 · Abstract (English)

Deep neural networks are increasingly bottlenecked by the cost of optimization, both in terms of GPU memory and compute time. Existing acceleration techniques, such as mixed precision, second-order methods, and batch size scaling, are typically used in isolation. We present Tri-Accel, a unified optimization framework that co-adapts three acceleration strategies along with adaptive parameters during training: (1) Precision-Adaptive Updates that dynamically assign mixed-precision levels to layers based on curvature and gradient variance; (2) Sparse Second-Order Signals that exploit Hessian/Fisher sparsity patterns to guide precision and step size decisions; and (3) Memory-Elastic Batch Scaling that adjusts batch size in real time according to VRAM availability. On CIFAR-10 with ResNet-18 and EfficientNet-B0, Tri-Accel achieves up to 9.9% reduction in training time and 13.3% lower memory usage, while improving accuracy by +1.1 percentage points over FP32 baselines. Tested on CIFAR-10/100, our approach demonstrates adaptive learning behavior, with efficiency gradually improving over the course of training as the system learns to allocate resources more effectively. Compared to static mixed-precision training, Tri-Accel maintains 78.1% accuracy while reducing memory footprint from 0.35GB to 0.31GB on standard hardware. The framework is implemented with custom Triton kernels, whose hardware-aware adaptation enables automatic optimization without manual hyperparameter tuning, making it practical for deployment across diverse computational environments. This work demonstrates how algorithmic adaptivity and hardware awareness can be combined to improve scalability in resource-constrained settings, paving the way for more efficient neural network training on edge devices and cost-sensitive cloud deployments.

优化框架混合精度显存优化自适应训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。