arXiv:2603.21418cs.CLcs.AI2026-03

用高效微调技术让葡萄牙语问答模型更快更省资源。

Efficient Fine-Tuning Methods for Portuguese Question Answering: A Comparative Study of PEFT on BERTimbau and Exploratory Evaluation of Generative LLMs

  • 用四种参数高效微调方法测试两种大小的BERTimbau模型。
  • LoRA仅需26.5%训练时间就达到95.8%基准性能,F1达81.32。
  • 小模型效率远超生成式大模型,更符合绿色AI理念。

尽管大型语言模型改变了自然语言处理,但其计算成本对低资源语言如巴西葡萄牙语构成访问障碍。本文系统评估了在巴西葡萄牙语问答数据集SQuAD-BR上,针对BERTimbau模型的参数高效微调(PEFT)与量化技术。共测试40种配置,涵盖四种PEFT方法(LoRA、DoRA、QLoRA、QDoRA)及两个模型规模(Base: 110M,Large: 335M)。结果揭示三大关键发现:(1) LoRA在BERTimbau-Large上实现95.8%基准性能,训练时间减少73.5%(F1=81.32 vs 84.86);(2) 提高学习率至2e-4可显著提升性能,最高带来+19.71 F1点增益;(3) 大模型量化韧性更强,损失仅4.83点(小模型为9.56点)。此外,对Tucano和Sabiá的探索性评估表明,生成式模型虽能通过LoRA达成竞争性F1分数,但所需GPU内存最多达BERTimbau-Base的4.2倍,训练时间多3倍,凸显编码器架构在该任务中的效率优势。

原文摘要 · Abstract (English)

Although large language models have transformed natural language processing, their computational costs create accessibility barriers for low-resource languages such as Brazilian Portuguese. This work presents a systematic evaluation of Parameter-Efficient Fine-Tuning (PEFT) and quantization techniques applied to BERTimbau for Question Answering on SQuAD-BR, the Brazilian Portuguese translation of SQuAD v1. We evaluate 40 configurations combining four PEFT methods (LoRA, DoRA, QLoRA, QDoRA) across two model sizes (Base: 110M, Large: 335M parameters). Our findings reveal three critical insights: (1) LoRA achieves 95.8\% of baseline performance on BERTimbau-Large while reducing training time by 73.5\% (F1=81.32 vs 84.86); (2) higher learning rates (2e-4) substantially improve PEFT performance, with F1 gains of up to +19.71 points over standard rates; and (3) larger models show twice the quantization resilience (loss of 4.83 vs 9.56 F1 points). These results demonstrate that encoder-based models can be efficiently fine-tuned for extractive Brazilian Portuguese QA with substantially lower computational cost than large generative LLMs, promoting more sustainable approaches aligned with \textit{Green AI} principles. An exploratory evaluation of Tucano and Sabiá on the same extractive QA benchmark shows that while generative models can reach competitive F1 scores with LoRA fine-tuning, they require up to 4.2$\times$ more GPU memory and 3$\times$ more training time than BERTimbau-Base, reinforcing the efficiency advantage of smaller encoder-based architectures for this task.

葡萄牙语微调绿色AIBERTimbau

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。