量化适配器能写多少信息？发现其记忆能力远低于参数数量预测。
How Many Bits Can an Adapter Write? Measuring the Capacity and Memorization of Parameter-Efficient Fine-Tuning
- 用比特直接测量适配器在冻结模型中写入的信息量
- 每可训练参数仅存储约2比特，且受位置影响显著
- 适用于评估微调隐私风险，尤其对监督与强化学习有区分意义
LoRA适配器通常以几兆字节大小被视作技能而非数据记录。我们将其假设置于可量化的尺度上。将基于压缩的记忆分析扩展至冻结主干设置，直接以比特为单位测量低秩适配器在未修改的模型中写入的内容。结果表明，其容量既小于全量微调，也低于参数计数所预测的规律性：每可训练参数仅存储约2比特，远低于完整模型的预算，且这一数值更取决于参数位置而非数量本身——将相同参数预算从注意力层移到MLP层，容量几乎翻倍；若移除冻结主干的结构，容量几乎消失。应用于Qwen2.5的真实微调任务中，同一工具显示隐私泄露随适配器写入的比特数上升，而非其名义参数量；并清晰划分了监督学习与强化学习：监督微调会逐字复制秘密，而基于可验证奖励训练的适配器则从不记录。直接测量微调写入内容，而非事后攻击，使一种坊间说法变为可设计应对的量化指标。
原文摘要 · Abstract (English)
A LoRA adapter is a few megabytes that almost everyone treats as a skill rather than a record of the data behind it. We put that assumption on a scale. Extending compression-based memorization analysis to the frozen-base setting, we measure directly, in bits, how much a low-rank adapter writes into a model it never changes. The answer is both smaller than full fine-tuning and less lawful than parameter counting would predict. Adapters store a couple of bits per trainable parameter, well short of a full model's budget, but that figure turns less on how many parameters an adapter carries than on where they sit. Move the same parameter budget from attention into the MLP and it holds nearly twice as much; strip the frozen base of its structure and the capacity all but disappears. Applied to realistic fine-tunes of Qwen2.5, the same instrument shows privacy leakage rising with the bits an adapter writes rather than the parameters it nominally has, and it draws a clean line between supervised and reinforcement learning: the secrets that supervised fine-tuning copies down verbatim, an adapter trained on verifiable rewards never records. Measuring what fine-tuning writes, rather than attacking it after the fact, turns a piece of folklore into a quantity one can design against.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。