用更细粒度的参数步长研究低秩适配器，发现效果因任务而异。
Finer Parameter Steps for Low-Rank PEFT: A Controlled Study with CP Tensor Adapters
- 采用CP张量分解实现每步仅193个可训练参数的精细调整
- 在BoolQ任务上优于LoRA，SST-2早期就达性能瓶颈
- 适合需要精准评估预算敏感性的模型微调研究者
低秩适配器通常通过少量秩值进行比较，但秩也决定了参数预算的分辨率。对于一个2048×2048的OPT注意力投影，每增加一个LoRA秩会引入4096个可训练标量，导致低预算适配器尺寸之间存在较大间隔。本文提出：使用更细粒度的张量化适配器是否会改变观测到的准确率-预算权衡？我们以固定组件的正则化CP张量适配器为例，在32×64×32×64张量化下，每个归一化CP成分每投影仅存储193个可训练标量，约为一个LoRA秩步长的1/21。我们在OPT-1.3B模型上，对SST-2、RTE和BoolQ任务，在匹配目标模块、训练协议、数据上限和种子调度的前提下，对比了CP适配器与LoRA。结果表明，CP适配器训练稳定且填补了LoRA秩之间的空白，但效果任务依赖：SST-2在低预算时即达平台期，BoolQ在饱和前仍能从额外的CP成分中获益，略低于LoRA；RTE始终偏好LoRA。因此，更细的参数步长有助于诊断PEFT预算敏感性，但不保证整体更优的准确率-预算曲线。
原文摘要 · Abstract (English)
Low-rank adapters are usually compared by sweeping a small set of ranks, but the rank also fixes the resolution of the parameter budget. For a $2048{\times}2048$ OPT attention projection, increasing LoRA by one rank stores $4096$ trainable scalars, leaving large gaps between feasible low-budget adapter sizes. This paper asks whether a tensorized adapter with finer capacity increments changes the observed accuracy--budget trade-off. We instantiate this question with fixed-component canonical polyadic (CP) tensor adapters. Under a $32{\times}64{\times}32{\times}64$ tensorization, one normalized CP component stores $193$ trainable scalars per projection, about $21$ times smaller than one LoRA rank step. We compare CP adapters and LoRA on OPT-1.3B across SST-2, RTE, and BoolQ under matched target modules, training protocol, data caps, and seed schedules. CP trains stably and fills the gaps between LoRA ranks, but the effect is task-dependent: SST-2 reaches an early low-budget plateau, BoolQ benefits from additional CP components before saturating slightly below LoRA, and RTE remains LoRA-favored. Finer parameter steps are therefore useful for diagnosing PEFT budget sensitivity, but they do not by themselves guarantee a better accuracy--budget curve.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。