arXiv:2508.17250cs.CLcs.IR2025-08被引 3

通过专家路由机制,让大模型高效生成商品组合。

Routing Distilled Knowledge via Mixture of LoRA Experts for Large Language Model based Bundle Generation

  • 用两种知识类型+LoRA专家混合架构,避免知识冲突。
  • 在三个数据集上准确率超教师模型,推理速度更快。
  • 适合需要高效生成商品组合的推荐系统开发者。

大型语言模型(LLMs)在自动商品组合生成中展现出潜力,但存在计算成本过高的问题。尽管知识蒸馏可提升学生模型效率,但初步研究发现,直接融合教师模型中的多种蒸馏知识会引发知识冲突,影响生成性能。为此,本文提出RouteDK框架,通过混合LoRA专家结构实现知识路由。首先,从教师模型中蒸馏出两类互补知识:高层级知识(通用规则)与细粒度知识(会话特定推理)。随后,为每类知识训练专属的LoRA专家,并与基础LoRA专家共同训练。为实现有效融合,设计了输入感知的动态融合模块,路由器根据输入动态分配专家权重,缓解知识冲突。为进一步提升推理可靠性,引入推理时增强模块以降低方差并抑制次优推理。在三个公开数据集上的实验表明,RouteDK在准确率上达到或优于教师模型,同时保持强计算效率,并显著优于当前最优的组合生成方法。

原文摘要 · Abstract (English)

Large Language Models (LLMs) have shown potential in automatic bundle generation but suffer from prohibitive computational costs. Although knowledge distillation offers a pathway to more efficient student models, our preliminary study reveals that naively integrating diverse types of distilled knowledge from teacher LLMs into student LLMs leads to knowledge conflict, negatively impacting the performance of bundle generation. To address this, we propose RouteDK, a framework for routing distilled knowledge through a mixture of LoRA expert architecture. Specifically, we first distill knowledge from the teacher LLM for bundle generation in two complementary types: high-level knowledge (generalizable rules) and fine-grained knowledge (session-specific reasoning). We then train knowledge-specific LoRA experts for each type of knowledge together with a base LoRA expert. For effective integration, we propose a dynamic fusion module, featuring an input-aware router, where the router balances expert contributions by dynamically determining optimal weights based on input, thereby effectively mitigating knowledge conflicts. To further improve inference reliability, we design an inference-time enhancement module to reduce variance and mitigate suboptimal reasoning. Experiments on three public datasets show that our RouteDK achieves accuracy comparable to or even better than the teacher LLM, while maintaining strong computational efficiency. In addition, it outperforms state-of-the-art approaches for bundle generation.

大模型知识蒸馏组合生成LoRA

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。