通过细粒度剪枝，大幅减少大模型推荐系统参数量，性能损失低于12%。
Boosting Parameter Efficiency in LLM-Based Recommendation through Sophisticated Pruning
- 发现并利用自注意力与MLP模块内的层内冗余，实现更精细剪枝。
- 在三个数据集上剪掉超95%非嵌入参数，仍保持原模型88%性能。
- 三阶段渐进式剪枝+蒸馏恢复，适合资源受限场景部署。
基于大语言模型的推荐系统虽取得显著进展，但其庞大的参数量带来高昂部署成本。本文探索参数剪枝以提升参数效率，同时保持推荐质量，推动实际应用。不同于以往聚焦层间冗余的方法，我们发现自注意力和MLP模块内部存在层内冗余。基于此，提出融合层内与层间剪枝的细粒度方法，采用三阶段策略：从层内到层间、从宽度到深度逐步剪枝，并在每阶段引入蒸馏技术恢复性能。实验表明，在三个数据集上，模型剪除超过95%的非嵌入参数后，仍可达到原始模型平均88%的性能表现，证明该方法能显著降低资源消耗而几乎不牺牲推荐效果。代码将公开于：https://github.com/zheng-sl/PruneRec。
原文摘要 · Abstract (English)
LLM-based recommender systems have made significant progress; however, the deployment cost associated with the large parameter volume of LLMs still hinders their real-world applications. This work explores parameter pruning to improve parameter efficiency while maintaining recommendation quality, thereby enabling easier deployment. Unlike existing approaches that focus primarily on inter-layer redundancy, we uncover intra-layer redundancy within components such as self-attention and MLP modules. Building on this analysis, we propose a more fine-grained pruning approach that integrates both intra-layer and layer-wise pruning. Specifically, we introduce a three-stage pruning strategy that progressively prunes parameters at different levels and parts of the model, moving from intra-layer to layer-wise pruning, or from width to depth. Each stage also includes a performance restoration step using distillation techniques, helping to strike a balance between performance and parameter efficiency. Empirical results demonstrate the effectiveness of our approach: across three datasets, our models achieve an average of 88% of the original model's performance while pruning more than 95% of the non-embedding parameters. This underscores the potential of our method to significantly reduce resource requirements without greatly compromising recommendation quality. Our code will be available at: https://github.com/zheng-sl/PruneRec
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。