用超网络生成LoRA适配器，让小模型更可信地预测概率。
HypeLoRA: Hyper-Network-Generated LoRA Adapters for Calibrated Language Model Fine-Tuning
- 用共享超网络动态生成LoRA的矩阵，实现层间结构耦合。
- 在GLUE上比全微调更准且校准度更高，尤其在CoLA上MCC更好。
- 适合追求高可靠性与参数效率的模型部署场景。
现代基于Transformer的模型常出现校准不足，产生过度自信的预测结果。本文研究了低秩适配（LoRA）在RoBERTa上的校准特性，并提出一种基于超网络的新型参数高效适配框架。在GLUE基准测试中，LoRA方法在保持显著参数效率的同时，校准性能与全微调相当，甚至在某些任务上更优。进一步引入共享超网络动态生成LoRA的A、B矩阵，实现跨层结构耦合，其表现接近标准LoRA，且在CoLA数据集上取得更高的马修相关系数（MCC）。研究还揭示关键权衡：限制适配空间（如冻结矩阵A）可有效降低期望校准误差（ECE），但需以牺牲下游任务准确率为代价。为支持后续研究，我们提供了统一可复现的校准评估代码，包含ECE、MCE和ACE等指标。研究澄清了参数效率与概率可靠性之间的关系，为不确定性感知的Transformer架构奠定基础。代码已开源。
原文摘要 · Abstract (English)
Modern Transformer-based models frequently suffer from miscalibration, producing overconfident predictions that do not reflect true empirical frequencies. This work investigates the calibration dynamics of LoRA: Low-Rank Adaptation and a novel hyper-network-based adaptation framework as parameter-efficient alternatives to full fine-tuning for RoBERTa. Evaluating across the GLUE benchmark, we demonstrate that LoRA-based adaptation consistently achieves calibration parity with (and in specific tasks exceeds) full fine-tuning, while maintaining significantly higher parameter efficiency. We further explore a dynamic approach where a shared hyper-network generates LoRA factors (A and B matrices) to induce structural coupling across layers. This approach produced results similar to standard LoRA fine-tuning, even achieving better MCC on CoLA dataset. Our study also reveal a critical trade-off: constraining the adaptation space (e.g., freezing matrices A) acts as a powerful regularizer that enhances Expected Calibration Error (ECE), but necessitates a carefully balanced sacrifice in downstream task accuracy. To support future research, we provide a unified and reproducible implementation of contemporary calibration metrics, including ECE, MCE, and ACE. Our findings clarify the relationship between parameter efficiency and probabilistic reliability, positioning structured low-rank updates as a viable foundation for uncertainty-aware Transformer architectures. Code available at: https://github.com/btrojan-official/HypeLoRA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。