arXiv:2607.22251cs.LGcs.AI2026-07

基于信息流拓扑的低秩微调,让参数分配更智能。

IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning

论文配图:IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning
图 1 · 摘自论文原文
  • 通过干预追踪构建任务相关的模块交互图,获取全局信息流动先验。
  • 在GSM8K上比标准LoRA高4.75个百分点,且无需额外训练开销。
  • 适合追求高效微调的开发者,尤其适用于资源受限场景。

低秩适应(LoRA)是大模型参数高效微调的常用方法,其效果依赖于秩的分配策略。现有自适应方法基于局部梯度、激活或矩阵统计生成秩,但训练时引入开销,且局部信号难以反映模块在信息传播中的结构角色,导致稀缺容量分配缺乏全局依据。本文提出IFCLoRA,一种用于预微调阶段秩分配与适配器初始化的拓扑感知方法。利用小规模校准集,IFCLoRA对冻结模型执行干预追踪,构建稀疏的任务条件交互图,从中提取全局信息流拓扑先验,并融合节点局部梯度敏感性,形成主导拓扑的信息流中心性(IFC)得分,衡量模块在任务相关多跳传播中的参与度。该得分作为模块级路由信号,在秩预算约束下实现一次性离线离散秩分配。同时,复用追踪中响应向量构造保持函数特性的流响应子空间初始化,使适配器具备任务相关输出子空间。在所有设置下,IFCLoRA均达到高于标准LoRA的平均得分,且微调时间与峰值内存相当;仅需一次离线校准。在GSM8K上,其平均准确率超过对比的其他PEFT方法,相较标准LoRA提升4.75个百分点,适用于基础模型如LLaMA-3.1-8B。结果表明,秩分配非均匀且随任务与模型变化,说明任务条件下的全局信息流拓扑可作为低预算微调中秩分配的有效结构先验。

原文摘要 · Abstract (English)

Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT) of LLMs whose effectiveness depends on rank allocation. Existing adaptive LoRA methods derive ranks from local gradient, activation, or matrix statistics collected before or during fine-tuning; training-time variants add overhead, and local signals reveal little about each module's structural role in information propagation, giving weak global grounding for scarce-capacity allocation. We propose IFCLoRA, a topology-aware method for pre-fine-tuning rank allocation and adapter initialization. Using a small calibration set, IFCLoRA performs intervention tracing on the frozen model and constructs a sparse task-conditioned interaction graph over LoRA target modules. From this graph it extracts a global information-flow topology prior and fuses it with each node's local gradient sensitivity to form a topology-dominant Information-Flow Centrality (IFC) score, measuring participation in task-conditioned multi-hop propagation. The IFC scores then serve as module-level routing signals for one-shot discrete rank allocation under a rank-budget constraint. Reusing response vectors from tracing, IFCLoRA constructs a function-preserving flow-response subspace initialization, giving adapters task-relevant output subspaces. Across all settings, IFCLoRA achieves higher mean scores than standard LoRA with comparable fine-tuning time and peak memory; it requires a one-time offline calibration stage. On GSM8K, IFCLoRA attains the highest mean accuracy among compared PEFT methods on both base models, exceeding standard LoRA by 4.75 percentage points on LLaMA-3.1-8B. Resulting rank allocations are non-uniform and vary across tasks and base models, suggesting that task-conditioned global information-flow topology can serve as a useful structural prior for rank allocation in low-budget PEFT.

参数高效微调信息流拓扑感知低秩适应

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。