arXiv:2504.15777cs.CLcs.AI2025-04被引 28

用极低成本实现强推理能力,仅需9美元就超越大模型表现。

Tina: Tiny Reasoning Models via LoRA

  • 基于1.5B小模型,通过LoRA进行高效强化学习微调。
  • 在AIME24上达43.33%准确率,性能超主流模型且成本降260倍。
  • 适合追求低成本高效率推理模型的研究者与开发者。

如何以低代价实现语言模型的强推理能力?我们提出Tina,一类通过参数高效强化学习(RL)实现的微型推理模型。Tina仅使用1.5B参数的小型基础模型,结合低秩适应(LoRA)技术,在强化学习中进行轻量级参数更新,即可获得媲美甚至超越现有最先进(SOTA)RL推理模型的性能。关键在于,其后训练与评估成本仅为9美元,相较现有方法降低约260倍。最优Tina模型在AIME24数据集上达到43.33%的Pass@1准确率,推理性能提升超过20%。我们在多个开源推理数据集及多种消融设置下验证了该方法的有效性,均采用单一固定超参数。我们推测,其成功源于LoRA能快速适配强化学习所奖励的推理结构,同时保留基础模型的知识。为促进开放研究,所有代码、训练日志及模型权重均已开源。

原文摘要 · Abstract (English)

How cost-effectively can strong reasoning abilities be achieved in language models? Driven by this fundamental question, we present Tina, a family of tiny reasoning models achieved with high cost-efficiency. Notably, Tina demonstrates that substantial reasoning performance can be developed using only minimal resources, by applying parameter-efficient updates during reinforcement learning (RL), using low-rank adaptation (LoRA), to an already tiny 1.5B parameter base model. This minimalist approach produces models that achieve reasoning performance which is competitive with, and sometimes surpasses, SOTA RL reasoning models built upon the same base model. Crucially, this is achieved at a tiny fraction of the computational post-training cost employed by existing SOTA models. In fact, the best Tina model achieves a >20\% reasoning performance increase and 43.33\% Pass@1 accuracy on AIME24, at only \$9 USD post-training and evaluation cost (i.e., an estimated 260x cost reduction). Our work reveals the surprising effectiveness of efficient RL reasoning via LoRA. We validate this across multiple open-source reasoning datasets and various ablation settings starting with a single, fixed set of hyperparameters. Furthermore, we hypothesize that this effectiveness and efficiency stem from LoRA rapidly adapting the model to the structural format of reasoning rewarded by RL, while largely preserving the base model's underlying knowledge. In service of accessibility and open research, we fully open-source all code, training logs, and model weights \& checkpoints.

推理模型LoRA低成本微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。