微小模型通过分类头微调，在多项选择任务上表现超越大模型。
Punching Above Their Weight: Classification-Head Fine-Tuning of Tiny Language Models (TLMs) for Verifiable Multiple-Choice Tasks
- 用判别式分类头替代生成标签，提升小模型性能
- 0.6B和1.7B模型在多个任务上提升2-3%，达顶尖水平
- 适合资源有限但需高精度推理的场景
我们将参数量低于约30亿的模型定义为微型语言模型（TLM），其可在主流消费级设备上运行。研究如何将其适配并应用于可验证的多项选择任务。在多个Qwen3模型（0.6B至8B）和五个基准测试集（HellaSwag、WinoGrande、PIQA、SciQ、ARC-C）上，对比了三种基于LoRA的微调范式（标签生成、仅金标准、以及我们提出的判别式分类头）。结果表明，分类头微调在0.6B和1.7B规模下显著优于标签生成（提升2-3%）。进一步地，使用该方法微调的TLM在性能上已可与零样本/少样本的GPT-3（175B）、PaLM（540B）和GPT-4媲美。报告中Qwen3-0.6B和Qwen3-1.7B在HellaSwag、WinoGrande和PIQA上的表现达到当前最优（SOTA）。
原文摘要 · Abstract (English)
We define Tiny Language Models (TLMs) as models below roughly 3B parameters that fit on mainstream consumer devices. We study how to adapt them for and use them on verifiable multiple-choice tasks. We compare three LoRA-based fine-tuning paradigms (label generation, gold only, and our discriminative classification head) on a unified setup across several Qwen3 models from 0.6B to 8B and five benchmarks: HellaSwag, WinoGrande, PIQA, SciQ and ARC-C. Classification-head fine-tuning reliably outperforms label generation (+2-3%) at the 0.6B and 1.7B scales. Further, TLMs fine-tuned using the discriminative method are competitive to zero-/few-shot GPT-3 (175B), PaLM (540B) and GPT-4. The performance we report for Qwen3-0.6B and Qwen3-1.7B are SOTA on HellaSwag, WinoGrande, and PIQA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。