arXiv:2607.27506cs.CLcs.AI2026-07

小模型也能高效做RAG,无需大预训练。

Models for minimalist RAG: B1ade 335M Embedding and 1B Parameter Small Language Models

论文配图:Models for minimalist RAG: B1ade 335M Embedding and 1B Parameter Small Language Models
图 1 · 摘自论文原文
  • 用5个预训练编码器融合出335M嵌入模型,零训练即达顶尖性能。
  • 10亿参数小语言模型在问答任务中准确率超80%,引用率达42.4%。
  • 强化学习让模型自发引用来源,适合资源有限的部署场景。

RAG系统中的语言与嵌入模型通常依赖大规模预训练和显式对齐监督。本文提出B1ade,一种高效RAG架构,包含两个专为任务设计的组件:一个335M参数的检索模型B1ade-embed和一个10亿参数的小语言模型B1ade-1B。B1ade-embed通过五种预训练编码器的无参融合构建,在不进行额外训练的情况下达到子5亿参数模型中的最高MTEB得分。B1ade-1B基于7.23亿词元(220万样本)的精选上下文-问题对,在低成本GPU上使用分组相对策略优化(GRPO)训练,奖励仅针对答案相似性。核心发现为涌现溯源:尽管未接受显式引用监督,其42.4%的回答会引用检索段落,比训练分布高出5.5个百分点。这表明在强化学习下,溯源可作为提升准确性的自然策略出现。在标准问答基准上,B1ade-1B在PopQA达81.82%,PubMedQA达65.8%,FEVER达51.09%。端到端RAG评估中,平均得分0.654,在正确性、完整性、连贯性和忠实性方面相比监督微调提升10.8%,并缩小了与1.5倍规模模型的差距。结果表明,通过合理模型组合与奖励设计,无需大规模预训练即可实现高效RAG。

原文摘要 · Abstract (English)

Language and embedding models used in RAG systems are conventionally assumed to require large-scale pretraining and explicit grounding supervision. We present B1ade, an efficient RAG architecture comprising two purpose-built components: a compact embedding model and a purpose-built SLM. B1ade-embed, a 335M parameter retrieval model constructed via parameter-free fusion of five pretrained encoders achieves top MTEB scores among sub-500M models with zero additional training, and B1ade-1B, an SLM trained on low-cost GPUs using Group Relative Policy Optimization (GRPO) on 723M tokens (2.2M examples) of curated context-question pairs with rewards that optimize only answer similarity. Our central finding is emergent attribution: despite receiving no explicit supervision for source citation, B1ade-1B cites retrieved passages in 42.4% of responses, exceeding the attribution rate of its training distribution by 5.5 percentage points. This demonstrates that grounding behavior can emerge as an accuracy-maximizing strategy under RL training, without explicit reward engineering. On standard QA benchmarks, B1ade-1B achieves 81.82% on PopQA, 65.8% on PubMedQA, and 51.09% on FEVER. In end-to-end RAG evaluation, B1ade-1B achieves an average score of 0.654 across correctness, completeness, coherence, and faithfulness, a 10.8% improvement over the SFT, while closing the gap with models 1.5x its size. These results show that strategic model composition and reward design suffice for resource-efficient RAG, without large-scale pretraining.

RAG小模型强化学习嵌入模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。