轻量级推理让大模型写故事更省算力
Lightweight Latent Reasoning for Narrative Tasks
- 用轻量投影模块生成连续潜变量，跳过部分推理步骤
- 在剧情检测和章节生成任务中减少77%-92%的推理长度
- 可与强化学习结合，适合追求高效推理的应用场景
大型语言模型通过生成长链思维或‘推理轨迹’来完成复杂任务，这些轨迹作为生成输出的潜在变量。利用强化学习（RL）优化推理轨迹可提升预测答案的效果，但计算成本高昂，尤其在涉及大量文本检索与处理的叙事任务中。为此，我们提出LiteReason，一种可与标准词元采样交织、易于与RL技术结合的潜变量推理方法。LiteReason采用轻量级推理投影器模块，训练其生成连续潜变量以帮助模型‘跳过’部分推理步骤。在强化学习过程中，策略模型决定何时激活投影器，动态切换潜变量与离散推理模式。在剧情漏洞检测和书籍章节生成任务上的实验表明，该方法优于现有潜变量推理基线，并接近非潜变量强化学习训练的表现，同时将最终推理长度减少77%-92%。总体而言，LiteReason引导强化学习训练向性能-计算权衡曲线中更高效的区域收敛。
原文摘要 · Abstract (English)
Large language models (LLMs) tackle complex tasks by generating long chains of thought or "reasoning traces" that act as latent variables in the generation of an output given a query. A model's ability to generate such traces can be optimized with reinforcement learning (RL) to improve their utility in predicting an answer. This optimization comes at a high computational cost, especially for narrative-related tasks that involve retrieving and processing many tokens. To this end, we propose LiteReason, a latent reasoning method that can be interleaved with standard token sampling and easily combined with RL techniques. LiteReason employs a lightweight Reasoning Projector module, trained to produce continuous latent tokens that help the model 'skip' reasoning steps. During RL, the policy model decides when to activate the projector, switching between latent and discrete reasoning as needed. Experimental results on plot hole detection and book chapter generation show that our method outperforms latent reasoning baselines and comes close to matching non-latent RL training, while reducing final reasoning length by 77-92%. Overall, LiteReason guides RL training to a more efficient part of the performance-computation tradeoff curve.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。