用强化学习选最优代码草稿,既省 token 又提质量。
Reinforcement Learning-Guided Chain-of-Draft for Token-Efficient Code Generation
- 通过强化学习从多个草稿中挑选最佳方案,优化选择策略。
- 在 MBPP 等数据集上表现媲美标准提示,节省超 50% 令牌成本。
- 适合追求高效、低成本部署的代码生成应用场景。
大模型在代码生成中虽表面流畅,却难以完成需正确性和语义一致性的结构化推理任务。传统思维链(CoT)提示虽提升推理能力,但冗长低效;代码草稿链(CoD)更简洁,但因大模型随机性导致解质量不一,难以选出最优解。本文提出 MultiCoD,一个基于强化学习的框架,可从 CoD 生成的多个候选解中筛选最具潜力者。该方法采用策略引导提示促进多样化推理风格,并将解选择建模为上下文相关多臂赌博机问题。通过融合代码复杂度、推理结构与策略元数据等可解释特征的奖励函数,平衡正确性、效率与清晰度。在 MBPP、BigCodeBench、SWE-bench Verified 与 Defects4J 上的实验表明,MultiCoD 在多数情况下性能优于或媲美标准提示、CoT 与 CoD 基线,同时通过多候选设计仅对最终选定输出计费,用户成本降低超过 50%,响应质量提升,更具可持续性与可扩展性。代码已开源:https://anonymous.4open.science/r/MultiCoD。
原文摘要 · Abstract (English)
LLMs demonstrate surface-level fluency in code generation but struggle with structured reasoning tasks requiring correctness and semantic alignment. While Chain-of-Thought (CoT) prompting enhances reasoning through intermediate steps, it suffers from verbosity and inefficiency. Chain-of-Draft (CoD) prompting offers more concise reasoning, but the stochastic nature of LLMs produces varying solution quality, making optimal selection challenging. We propose \multicod, a reinforcement learning framework that learns to select the most promising candidate from CoD-generated solutions. Our approach uses strategy-guided prompting to encourage diverse reasoning styles and models solution selection as a contextual bandit problem. The framework optimizes interpretable features including code complexity, reasoning structure, and strategic metadata through a reward function balancing correctness, efficiency, and clarity. Experiments on MBPP, BigCodeBench, SWE-bench Verified, and Defects4J show \multicod~outperforms and in some cases, on par with standard prompting, CoT, and CoD baselines while achieving cost and token efficiency from the user's perspective through a multi-candidate design that charges only for the selected output, reducing user billing by over 50\% and improving LLM response quality, making \multicod~more sustainable and scalable for real-world deployment. Our code is available: https://anonymous.4open.science/r/MultiCoD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。