arXiv:2608.01624cs.CLcs.LG2026-08

调参不靠维度,而是看扰动幅度大小,决定语言模型微调效果。

Not the Dimension, the Norm: What Matters in Gradient-Free Weight Perturbation of Language Models

论文配图:Not the Dimension, the Norm: What Matters in Gradient-Free Weight Perturbation of Language Models
图 1 · 摘自论文原文
  • 固定评分和投票方式,只改变扰动的子空间与幅度
  • 仅扰动12~16个标量时,平均比全参数扰动低1.8准确率点
  • 关键在于扰动幅度范围,跨模型跨规模都稳定有效

将语言模型适配到任务不再需要训练全部权重,参数高效方法已将可训练参数从数十亿降至少数标量。但无梯度微调仍需扰动权重张量的所有元素,其必要性未知。现有方法同时改变搜索空间、扰动尺度和聚合方式,难以判断何因素真正起作用。本文在固定管道下逐项干预:保持候选评分与投票一致,仅变化搜索维度、扰动子空间及其范数。结果发现,仅扰动12至16个标量的冻结框架,平均比全权重扰动低1.8准确率点,且在49个模型-基准组合中36个落后。维度与基底选择均非关键因素——即使与SVD基底格拉斯曼重叠处于随机水平,只要单一尺度匹配,性能即相同;在大尺度下,SVD方向率先失效。真正决定成败的是扰动范数,其可用范围在七种模型间仅差五倍,且内部平坦。因此,扰动范数是唯一具失败模式的因子,其安全区间可跨尺度与模型家族迁移。设计问题由此从‘扰动哪个子空间’变为‘扰动多剧烈’。

原文摘要 · Abstract (English)

Adapting a language model to a task no longer requires training all of its weights, and a line of parameter-efficient methods has driven the trainable count from billions down to a handful of scalars. Gradient-free adaptation, which samples random weight perturbations and keeps the ones that score well, has not followed that trajectory and still perturbs every entry of the weight tensor. It is unknown whether that full-weight search is necessary, and more fundamentally which property of a perturbation makes it work at all, because existing methods vary the search space, the perturbation scale, and the aggregation together. We resolve this by intervening on one factor at a time inside a fixed pipeline, holding candidate scoring and voting constant while we vary the search dimension, the subspace that carries the perturbation, and its norm. Perturbing a frozen frame of 12 to 16 scalars stays 1.8 accuracy points behind full-weight search on average across 49 model-benchmark cells, trailing it in 36 of them. Neither the dimension nor the choice of basis explains that performance. A random frame whose Grassmann overlap with the SVD frame is at chance level performs identically once a single scale factor is matched, and at large scales the SVD directions collapse first. What survives is the perturbation norm, whose usable range closes within a factor of five across seven models and stays flat inside. The perturbation norm is therefore the one factor with a failure mode, and its safe region transfers across scale and family. The design question narrows from which subspace to perturb to how hard to shake.

参数高效无梯度优化扰动范数语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。