让语音大模型训练更精准，通过树状结构分配奖励。
LEAF: Growing Trees Without Branching for Speech-Aware Large Language Model Post-Training
- 用回溯式树结构重建响应中的共享前缀与分叉点
- 在相同计算预算下，问答与翻译任务均超越现有方法
- 小模型性能超全参数先进模型，适合资源受限场景
当前基于GRPO的语音感知大模型微调方法存在奖励分配粗糙的问题，将相同的终端奖励平均分配给响应中每个词元，忽略了回滚批次内语音条件生成中存在的共享前缀与关键分叉结构。本文提出低秩探索自适应分叉（LEAF），一种无需在线分叉或额外解码的回溯树形强化学习方法。LEAF采样完整响应，选取高突变性边界，按共享前缀聚类，并基于后代奖励分配段级优势。理论证明了其段级信用分配与边界选择设计的有效性。实验表明，在相同回滚与低秩适配预算下，LEAF在语音问答与语音翻译基准上均优于GRPO；值得注意的是，更小的LEAF训练模型性能超过当前最先进的全参数基线。
原文摘要 · Abstract (English)
State-of-the-art GRPO-style methods for speech-aware large language model post-training suffer from coarse credit assignment, broadcasting the same terminal-reward advantage to every token in a response. This ignores useful structure within rollout batches, where speech-conditioned completions often share prefixes before diverging at important decisions. We propose Low-rank Exploration with Adaptive Forking (LEAF), a retrospective tree-based RL method that recovers this structure without online branching or additional decoding. LEAF samples complete responses, selects high-surprisal boundaries, groups responses by shared prefixes, and assigns span-level advantages using descendant rewards. We theoretically justify LEAF's span-level credit assignment and boundary-selection design. Empirically, LEAF improves over GRPO across speech question answering and speech translation benchmarks under the same rollout and low-rank adaptation budget. Notably, smaller LEAF-trained models outperform current state-of-the-art, full-parameter baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。