arXiv:2504.19583cs.LGcs.CL2025-04被引 8

用图谱分解优化大模型微调参数协同,提升效率与结构感知能力。

Graph-Based Spectral Decomposition for Parameter Coordination in Language Model Fine-Tuning

  • 将模型参数建模为图节点,通过拉普拉斯谱分解实现参数空间的频域建模。
  • 引入谱正则化与滤波机制,使梯度更新更稳定,收敛更快且参数扰动更小。
  • 适用于高效微调场景,尤其适合追求训练稳定性与泛化性能的用户。

本文提出一种基于图谱分析的参数协同优化算法,用于大语言模型的微调。将预训练模型的参数视为图中的节点,构建加权图并进行拉普拉斯谱分解,实现参数空间的频域建模与结构表征。在此基础上设计联合损失函数,结合任务损失与谱正则项,促进参数间的协同更新。同时,在优化阶段引入谱滤波机制,以结构感知方式调整梯度,增强训练稳定性和收敛性。在多个任务上评估,包括传统微调对比、少样本泛化测试和收敛速度分析,结果均显示该方法表现更优。实验验证了谱协同优化框架能有效降低参数扰动,提升微调质量,同时保持整体模型性能。该工作推动了大规模模型参数高效训练方法的发展,强调了结构信号处理在深度学习优化中的重要性,并提供了一个鲁棒且可推广的框架,以增强语言模型的适应性与性能。

原文摘要 · Abstract (English)

This paper proposes a parameter collaborative optimization algorithm for large language models, enhanced with graph spectral analysis. The goal is to improve both fine-tuning efficiency and structural awareness during training. In the proposed method, the parameters of a pre-trained language model are treated as nodes in a graph. A weighted graph is constructed, and Laplacian spectral decomposition is applied to enable frequency-domain modeling and structural representation of the parameter space. Based on this structure, a joint loss function is designed. It combines the task loss with a spectral regularization term to facilitate collaborative updates among parameters. In addition, a spectral filtering mechanism is introduced during the optimization phase. This mechanism adjusts gradients in a structure-aware manner, enhancing the model's training stability and convergence behavior. The method is evaluated on multiple tasks, including traditional fine-tuning comparisons, few-shot generalization tests, and convergence speed analysis. In all settings, the proposed approach demonstrates superior performance. The experimental results confirm that the spectral collaborative optimization framework effectively reduces parameter perturbations and improves fine-tuning quality while preserving overall model performance. This work contributes significantly to the field of artificial intelligence by advancing parameter-efficient training methodologies for large-scale models, reinforcing the importance of structural signal processing in deep learning optimization, and offering a robust, generalizable framework for enhancing language model adaptability and performance.

大模型微调图神经网络谱方法参数优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。