针对手机端小模型个性化，找到省电省内存的最佳微调方法。
Energy- and Memory-Efficient PEFT Methods for Personalized On-Device SLMs on Consumer GPUs
- 对比五种微调方法在两类小模型上的表现，考虑能耗与显存
- LoRA+在19组配置中能耗最优，QLoRA显存占用减少3.9倍
- 根据能耗或显存瓶颈选择对应最佳方法，适配手机等设备
尽管大语言模型发展迅速，但在资源受限设备上部署和个性化仍因高显存、耗时和高能耗难以实现。参数高效微调（PEFT）的小语言模型（SLMs）提供了可行替代方案，但很少有研究在多种架构下同时使用通用和个性化基准，并考虑能耗。我们比较了五种微调方法（全量微调、LoRA、LoRA+、QLoRA、BitFit）在四款来自两类架构（Transformer：TinyLlama-1.1B、Qwen3-1.7B；SSM：Mamba-1.4B、Mamba-2-1.3B）的SLMs上，在三个GLUE任务（SST-2、QNLI、STS-B）和三个LaMP个性化任务（LaMP-1、LaMP-2、LaMP-3）的表现。每种配置均以能量导向的NetScore-E和内存导向的NetScore-M评估，反映设备部署约束。方法选择遵循严格的能耗优先规则（最高NetScore-E，平局时看NetScore-M）。LoRA+在24组配置中19次取得最高NetScore-E，13次最高NetScore-M，且在18组中被选为最优。QLoRA仅适用于Transformer模型，相比LoRA将峰值微调显存降低最多3.9倍，因此在12组变压器配置中5次获得最佳NetScore-M，但因解量化开销，一旦考虑能耗，仅一次被选中。BitFit和全量微调几乎从未胜出。TinyLlama-1.1B在六项测试中五项能源表现最优，四项内存表现最优。结果表明，结合紧凑型SLMs与PEFT，可实现节能感知的个性化本地部署，最优方法由主导约束决定：能耗优先选LoRA+，内存优先选QLoRA。
原文摘要 · Abstract (English)
Despite rapid advances in large language models (LLMs), deploying and personalizing them on resource-constrained devices remains impractical due to high VRAM, time, and energy costs. Parameter-Efficient Fine-Tuning (PEFT) of Small Language Models (SLMs) offers a promising alternative, yet few studies compare PEFT methods across architectures using both general and personalization benchmarks while accounting for energy consumption. We compare five fine-tuning approaches (Full Fine-Tuning, LoRA, LoRA+, QLoRA, and BitFit) on four SLMs from two families (Transformer-based: TinyLlama-1.1B, Qwen3-1.7B; SSM-based: Mamba-1.4B, Mamba-2-1.3B) across three GLUE tasks (SST-2, QNLI, STS-B) and three LaMP personalization tasks (LaMP-1, LaMP-2, LaMP-3). Each configuration is evaluated with the energy-focused NetScore-E and the memory-focused NetScore-M, the two variants that reflect the constraints binding on-device deployment. Methods are selected with a strict energy-first rule (highest NetScore-E, ties broken by NetScore#). LoRA+ achieves the highest NetScore-E in 19 of 24 configurations and the highest NetScore-M in 13 of 24, and is the selected method in 18 of 24. QLoRA, available only for the Transformer models, cuts peak finetuning VRAM by up to 3.9x relative to LoRA and therefore takes the best NetScore-M in 5 of the 12 Transformer configurations, although its de-quantization overhead leaves it selected in only one of them once energy decides. BitFit and full fine-tuning are almost never competitive on either variant, and TinyLlama-1.1B leads the energy-focused NetScore-E on five of the six benchmarks and the memory-focused NetScore-M on four. These results show that compact SLMs paired with PEFT provide a practical, energy-aware path to personalized on-device deployment, with the optimal method set by the dominant constraint: LoRA+ for energy and QLoRA for memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。