LoRA权重隐含任务信息,可直接解析其行为与性能。
W2T: LoRA Weights Already Know What They Can Do
- 通过QR+SVD将LoRA权重映射到唯一标准形式,消除分解歧义。
- 在语言与视觉任务中,准确预测性能并检索适配器,准确率超90%。
- 无需运行模型或访问数据,适合高效模型分析与调试。
每个LoRA检查点以低秩权重矩阵紧凑存储特定任务的更新,为适配大语言模型至新任务和领域提供了高效方式。理论上,这些权重已编码适配器的功能及其性能表现。本文探究能否直接从权重中读取此类信息,而无需运行基础模型或访问训练数据。主要挑战在于单个LoRA更新存在无穷多种分解方式;若不解决此歧义,基于因子的模型可能拟合特定分解而非底层更新。为此,我们提出W2T,通过QR分解后接SVD,将每个LoRA更新映射至可证明唯一的标准形式,使所有等价分解共享同一表示。所得组件随后被分词并由Transformer处理,生成权重空间嵌入。在多个语言与视觉LoRA集合上,W2T在属性分类、性能预测与适配器检索任务中均取得优异表现,证明一旦消除分解歧义,LoRA权重能可靠指示模型行为。代码已公开于https://github.com/xiaolonghan2000/Weight2Token。
原文摘要 · Abstract (English)
Each LoRA checkpoint compactly stores task-specific updates in low-rank weight matrices, offering an efficient way to adapt large language models to new tasks and domains. In principle, these weights already encode what the adapter does and how well it performs. In this paper, we ask whether this information can be read directly from the weights, without running the base model or accessing training data. A key obstacle is that a single LoRA update can be factorized in infinitely many ways. Without resolving this ambiguity, models trained on the factors may fit the particular factorization rather than the underlying update. To this end, we propose \methodfull, which maps each LoRA update to a provably canonical form via QR decomposition followed by SVD, so that all equivalent factorizations share the same representation. The resulting components are then tokenized and processed by a Transformer to produce a weight-space embedding. Across language and vision LoRA collections, W2T achieves strong results on attribute classification, performance prediction, and adapter retrieval, demonstrating that LoRA weights reliably indicate model behavior once factorization ambiguity is removed. Code is available at https://github.com/xiaolonghan2000/Weight2Token.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。