通过检索得分偏度实现无需训练的LLM路由，显著降低推理开销。
SkewRoute: Training-Free LLM Routing for Knowledge Graph Retrieval-Augmented Generation via Score Skewness of Retrieved Context
- 利用检索结果得分分布偏度判断查询难易程度，指导LLM选择
- 路由效果提升3倍以上，运行时间低于原方法0.001倍
- 无需训练、即插即用，适合知识图谱增强生成场景
大型语言模型在多项任务中表现优异，但部署时推理成本高。为缓解幻觉问题，许多系统采用知识图谱增强检索增强生成（KG-RAG）。然而大量检索上下文进一步增加推理开销。一种平衡性能与成本的可行方案是LLM路由：将简单查询导向小型模型，复杂查询导向大型模型。然而当前尚无专用于RAG的路由方法，且已有基于训练的路由器因需大量训练数据，在该领域难以扩展。我们观察到检索评分器产生的得分分布与查询难度强相关。基于此，提出首个专为KG-RAG设计的极简有效路由框架——SkewRoute，可无须训练地高效平衡性能与成本，且支持即插即用。实验表明其路由有效性超现有方法3倍以上，运行时间降至不足0.001倍。代码已开源。
原文摘要 · Abstract (English)
Large language models excel at many tasks but often incur high inference costs during deployment. To mitigate hallucination, many systems use a knowledge graph to enhance retrieval-augmented generation (KG-RAG). However, the large amount of retrieved knowledge contexts increase these inference costs further. A promising solution to balance performance and cost is LLM routing, which directs simple queries to smaller LLMs and complex ones to larger LLMs. However, no dedicated routing methods currently exist for RAG, and existing training-based routers face challenges scaling to this domain due to the need for extensive training data. We observe that the score distributions produced by the retrieval scorer strongly correlate with query difficulty. Based on this, we propose an extremely simple yet effective routing framework, the first specifically designed for KG-RAG that efficiently balances performance and cost in a plug-and-play manner. It delivers over 3x higher routing effectiveness while reducing runtime to less than 0.001x compared to existing methods. Our code is available at https://github.com/hrwang00/SkewRoute.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。