通过拓扑分析选择性冻结注意力矩阵,实现高效低资源大模型微调。
TopoTuner: Topological Finetuning of Large Language Models

- 基于拓扑距离判断注意力矩阵变化,自动识别可冻结层。
- 仅训练1-2%参数,性能接近全量微调,9个场景中7个优于LoRA。
- 可跨任务复用冻结策略,适合资源受限的模型部署场景。
全量微调虽有效但成本高,而LoRA虽减少参数量却无法指导哪些组件应训练、哪些可冻结。本文提出TopoTuner,一种基于拓扑引导的微调框架,通过计算投影矩阵的持久图间Wasserstein距离,衡量其在微调过程中的拓扑变化,从而生成可复用的冻结策略。该方法在源数据集上学习冻结模式,并迁移到下游数据集,在问答与情感分析任务上验证其泛化能力。在LLaMA-3.1-8B、Mistral-7B-v0.3和Qwen3-8B-Base上,仅更新1%-2%参数即可达到与全量微调相当的效果,且在9个模型-数据组合中7个优于LoRA,最多可调整39.57%的投影参数。相比全量微调,训练时间平均减少20.4%;相比LoRA,减少5.5%。该方法为跨任务复用冻结策略开辟新路径。
原文摘要 · Abstract (English)
Full fine-tuning remains a strong way to adapt pretrained LLMs, but it updates all weights and can be expensive. LoRA reduces the number of trainable parameters, but it does not directly answer which pretrained components should be trained and which can be frozen during adaptation. We introduce TopoTuner, a topology-guided fine-tuning framework for selective freezing of attention projection matrices. \method treats each projection matrix as a row cloud and uses Wasserstein distances between persistence diagrams to measure how its topology changes during fine-tuning. TopoTuner learns a reusable freezing profile from a source dataset and transfers it to efficiently fine-tune models on out-of-domain datasets, evaluating whether task-specific topological drift generalizes across question answering and sentiment analysis tasks. Across LLaMA-3.1-8B, Mistral-7B-v0.3, and Qwen3-8B-Base, TopoTuner is competitive with full fine-tuning while training only 1-2\% of the model parameters, and outperforms LoRA in 7 out of 9 model-dataset settings, which can change up to 39.57\% of the projection parameters. Along with minimized updates, TopoTuner reduces training time by 20.4\% relative to full fine-tuning and 5.5\% relative to LoRA on average. TopoTuner opens a new direction for reusable freezing profiles, where fine-tuning behavior learned on one dataset can be shared across multiple tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。