提出MARC方法,让大模型推荐表示更高效且效果更好。
Modular Representation Compression: Adapting LLMs for Efficient and Effective Recommendations
- 通过模块化调整与任务解耦,控制大模型内部结构
- 中间层表示优于末层,压缩后推荐效果提升2.82% eCPM
- 适合工业级推荐系统优化,尤其关注存储与计算效率
近期大语言模型(LLMs)推动了推荐系统(RSs)的发展,但主流方法将LLM离线生成高维表示并预缓存,带来巨大存储与计算开销。我们发现一个反直觉现象:中层表示优势(MRA),即LLM中间层表示在推荐任务中表现优于末层。这是因为末层被强制专注于代理训练任务,导致其功能退化。基于模块化理论,我们提出模块化表示压缩(MARC),通过模块化调整引入压缩与任务适配模块,使LLM专精于表示学习;再通过模块化任务解耦,用信息约束与不同网络结构实现任务分离。大量实验验证,MARC有效缓解MRA问题,生成高效表示。值得注意的是,在大规模商业搜索广告场景的线上A/B测试中,MARC实现了2.82%的eCPM提升。
原文摘要 · Abstract (English)
Recently, large language models (LLMs) have advanced recommendation systems (RSs), and recent works have begun to explore how to integrate LLMs into industrial RSs. While most approaches deploy LLMs offline to generate and pre-cache augmented representations for RSs, high-dimensional representations from LLMs introduce substantial storage and computational costs. Thus, it is crucial to compress LLM representations effectively. However, we identify a counterintuitive phenomenon during representation compression: Mid-layer Representation Advantage (MRA), where representations from middle layers of LLMs outperform those from final layers in recommendation tasks. This degraded final layer renders existing compression methods, which typically compress on the final layer, suboptimal. We interpret this based on modularity theory that LLMs develop spontaneous internal functional modularity and force the final layer to specialize in the proxy training task. Thus, we propose \underline{M}odul\underline{a}r \underline{R}epresentation \underline{C}ompression (MARC) to explicitly control the modularity of LLMs. First, Modular Adjustment explicitly introduces compression and task adaptation modules, enabling the LLM to operate strictly as a representation-learning module. Next, to ground each module to its specific task, Modular Task Decoupling uses information constraints and different network structures to decouple tasks. Extensive experiments validate that MARC addresses MRA and produces efficient representations. Notably, MARC achieved a 2.82% eCPM lift in an online A/B test within a large-scale commercial search advertising scenario.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。