针对日语小模型，系统优化了训练规模、模型选型与量化策略。
Adapting Methods for Domain-Specific Japanese Small LMs: Scale, Architecture, and Quantization
- 用QLoRA微调，实验确定4000样本为最佳训练量。
- 日语持续预训练的Llama-3模型性能优于多语言模型。
- 推荐使用Swallow-8B Q4_K_M，适合消费级设备部署。
本文提出一套构建领域专用日语小型语言模型的系统方法，采用QLoRA微调技术。研究解决三个核心问题：最优训练规模、基模型选择与架构感知量化。第一阶段（训练规模）：通过1,000至5,000样本的尺度学习实验，发现n=4,000时测试集负对数似然最低（1.127），在5,000样本时出现过拟合。第二阶段（模型对比）：对比四种日语大模型，经日语持续预训练的Llama-3模型（Swallow-8B、ELYZA-JP-8B）优于多语言模型（Qwen2.5-7B）。第三阶段（量化）：Llama-3架构在Q4_K_M量化下表现良好，而GQA架构严重退化（Qwen2.5下降0.280分）。生产推荐：Swallow-8B Q4_K_M实现2.830/3得分，单题响应时间8.9秒，模型大小4.9 GB。该方法可推广至低资源技术领域，为消费级硬件上的紧凑日语专用模型提供可操作指导。
原文摘要 · Abstract (English)
This paper presents a systematic methodology for building domain-specific Japanese small language models using QLoRA fine-tuning. We address three core questions: optimal training scale, base-model selection, and architecture-aware quantization. Stage 1 (Training scale): Scale-learning experiments (1k--5k samples) identify n=4,000 as optimal, where test-set NLL reaches minimum (1.127) before overfitting at 5k samples. Stage 2 (Compare finetuned SLMs): Comparing four Japanese LLMs shows that Llama-3 models with Japanese continual pre-training (Swallow-8B, ELYZA-JP-8B) outperform multilingual models (Qwen2.5-7B). Stage 3 (Quantization): Llama-3 architectures improve under Q4_K_M quantization, while GQA architectures degrade severely (Qwen2.5: -0.280 points). Production recommendation: Swallow-8B Q4_K_M achieves 2.830/3 score, 8.9 s/question, 4.9 GB size. The methodology generalizes to low-resource technical domains and provides actionable guidance for compact Japanese specialist LMs on consumer hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。