用小模型高效判断检索结果好坏，让智能体更快更省地决策。
Tiny-Critic RAG: Empowering Agentic Fallback with Parameter-Efficient Small Language Models
- 用低秩适配的小模型做快速判断，代替大模型做路由决策。
- 在噪声数据上准确率接近GPT-4o-mini，延迟降低十倍。
- 适合高并发、低成本部署的智能体系统，尤其看重响应速度。
检索增强生成（RAG）通过引入外部信息缓解大模型的幻觉问题。近期研究从静态流程转向模块化与自主式RAG框架，使模型具备多跳推理或自我修正能力。然而，现有反思式RAG严重依赖大型语言模型作为通用评估器。在高吞吐场景下，仅为了执行二值路由就运行百亿参数模型的完整前向传播，造成严重计算冗余。此外，在自主代理场景中，错误检索会导致模型浪费大量token进行无效推理和重复工具调用，显著增加首次响应时间（TTFT）和成本。我们提出Tiny-Critic RAG，通过低秩适配（LoRA）部署参数高效的小微语言模型（SLM），解耦评估环节。作为确定性门控机制，Tiny-Critic采用约束解码与无思考推理模式，实现极低延迟的二值路由。在注入噪声的数据集上的评估表明，其路由准确率可媲美GPT-4o-mini，同时延迟降低一个数量级，为智能体部署建立了一种高度经济高效的范式。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) grounds Large Language Models (LLMs) to mitigate factual hallucinations. Recent paradigms shift from static pipelines to Modular and Agentic RAG frameworks, granting models autonomy for multi-hop reasoning or self-correction. However, current reflective RAG heavily relies on massive LLMs as universal evaluators. In high-throughput systems, executing complete forward passes for billion-parameter models merely for binary routing introduces severe computational redundancy. Furthermore, in autonomous agent scenarios, inaccurate retrieval causes models to expend excessive tokens on spurious reasoning and redundant tool calls, inflating Time-to-First-Token (TTFT) and costs. We propose Tiny-Critic RAG, decoupling evaluation by deploying a parameter-efficient Small Language Model (SLM) via Low-Rank Adaptation (LoRA). Acting as a deterministic gatekeeper, Tiny-Critic employs constrained decoding and non-thinking inference modes for ultra-low latency binary routing. Evaluations on noise-injected datasets demonstrate Tiny-Critic RAG achieves routing accuracy comparable to GPT-4o-mini while reducing latency by an order of magnitude, establishing a highly cost-effective paradigm for agent deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。