用心理测量学方法精准匹配问题与大模型，兼顾效果与可解释性。
IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory
- 基于项目反应理论建模模型能力与问题难度的匹配关系。
- 在20个模型、12个数据集上优于多数基线方法，冷启动表现更优。
- 输出可解释的模型能力评分和问题难度评估，适合需要透明决策的场景。
大语言模型在各类自然语言任务中表现出色，但选择最优模型需在性能与成本间权衡：强模型效果好但昂贵，小模型便宜却能力弱。为此，本文提出 IRT-Router 框架，通过借鉴心理测量学中的项目反应理论（Item Response Theory），显式建模大模型能力与用户问题属性之间的关系。该方法不仅能准确预测响应性能，还能提供可解释的洞察，如模型能力与问题难度。此外，设计了一种基于语义相似性的在线查询预热技术，进一步提升模型在实际应用中的泛化能力。在20个大模型和12个数据集上的大量实验表明，IRT-Router 在有效性和可解释性方面均优于多数基线方法，尤其在冷启动场景下表现稳健，验证了其在真实场景中的可靠性与实用性。代码已开源。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated exceptional performance across a wide range of natural language tasks. However, selecting the optimal LLM to respond to a user query often necessitates a delicate balance between performance and cost. While powerful models deliver better results, they come at a high cost, whereas smaller models are more cost-effective but less capable. To address this trade-off, we propose IRT-Router, a multi-LLM routing framework that efficiently routes user queries to the most suitable LLM. Inspired by Item Response Theory (IRT), a psychological measurement methodology, IRT-Router explicitly models the relationship between LLM capabilities and user query attributes. This not only enables accurate prediction of response performance but also provides interpretable insights, such as LLM abilities and query difficulty. Additionally, we design an online query warm-up technique based on semantic similarity, further enhancing the online generalization capability of IRT-Router. Extensive experiments on 20 LLMs and 12 datasets demonstrate that IRT-Router outperforms most baseline methods in terms of effectiveness and interpretability. Its superior performance in cold-start scenarios further confirms the reliability and practicality of IRT-Router in real-world applications. Code is available at https://github.com/Mercidaiha/IRT-Router.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。