用扩散模型加速推理重排,速度提升2.4倍且准确率接近传统方法。
Diffusion-GR2: Diffusion Generative Reasoning Re-ranker
- 将自回归推理重排器转为块扩散模型,实现并行解码
- 通过自适应微调和在线蒸馏,修复生成无效排序的问题
- 适合需要高速推荐系统的场景,如电商搜索
生成式推理重排器通过输出思维链来提升推荐准确性,但推理速度慢:自回归解码器每生成一个推理词需一次顺序前向传播,推理过程远长于最终排序。为降低成本,块扩散语言模型可在少数去噪步骤内并行解码多个位置,显著提速。然而,直接转换自回归重排器会引发两个准确率差距:(1) 结构性差距——答案位置并行去噪且独立评分,导致生成重复、丢失或越界的标识符,而自回归模型通过左到右掩码避免此问题;(2) 分布差距——在固定教师轨迹上微调模型是离策略的,与自身推理分布不一致,留下残余误差。为此,我们提出 Diffusion-GR2,一套将自回归推理重排器(GR2)转化为块扩散重排器的方案。首先,转换微调(CFT)使初始自回归扩散模型能自行去噪生成有效排列,无需外部约束解码器;其次,基于策略蒸馏(OPD)利用自回归教师的密集逐标记目标,监督模型自身的解码轨迹;最后,在此基础上引入强化学习(RL)阶段,以重排奖励优化策略。在 Amazon Beauty 数据集上的实验表明,Diffusion-GR2 恢复至接近自回归重排器的性能,同时在推理输出长度下,解码吞吐量提升 2.4–3.5 倍。消融实验证明,CFT 恢复了大部分转换损失,而 OPD 进一步将其逼近自回归基准。
原文摘要 · Abstract (English)
Generative reasoning re-rankers achieve strong recommendation accuracy by emitting a chain-of-thought before re-ordering a candidate list, but they are slow at inference: an autoregressive (AR) decoder spends one sequential forward pass per reasoning token, and the reasoning trace far exceeds the ranking it produces. To reduce this cost, block-diffusion language models decode many positions in parallel over a few denoising steps and are substantially faster, yet naively converting an AR re-ranker into one opens two accuracy gaps: (1) a structural gap: answer positions are denoised in parallel and scored independently, so the decoder emits invalid rankings (duplicated, dropped, or out-of-set identifiers) that AR avoids through left-to-right masking; and (2) a distributional gap: fine-tuning the converted model on fixed teacher trajectories is off-policy relative to its own decoding at inference, leaving a residual accuracy gap. To close both gaps while keeping the speedup, we propose \textbf{Diffusion-GR2}, a recipe that converts our AR reasoning re-ranker (GR2) into a block-diffusion re-ranker. First, conversion fine-tuning (CFT) adapts the AR-initialized diffusion model to denoise the answer into a valid permutation on its own, without an external constrained decoder. Next, on-policy distillation (OPD) then supervises the model on its own decoded trajectories with dense per-token targets from the AR teacher. Finally, we apply a reinforcement-learning (RL) stage against a re-ranking reward on top of OPD's on-policy policy. Experiments on Amazon Beauty demonstrate that Diffusion-GR2 recovers to near-parity with the AR re-ranker, while block-parallel decoding raises decode throughput by $2.4$--$3.5\times$ at the model's reasoning output length. Ablations show that CFT recovers most of the conversion gap, and that on-policy distillation further closes it to the AR reference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。