提升大模型智能体效率，减少推理步骤与令牌消耗。
DEPO: Dual-Efficiency Preference Optimization for LLM Agents
- 提出双维度效率：每步耗令牌少、完成任务步数少。
- 实验显示令牌减少60.9%，步骤减少26.9%，性能提升29.3%。
- 适用于资源受限场景，且小样本训练仍有效。
大型语言模型在作为智能体部署时，推理能力显著增强，但更复杂的思维链（CoT）导致交互效率下降。目前缺乏对智能体效率的系统定义，制约了针对性优化。为此，本文提出双效率概念：（i）步骤级效率，即每步最小化令牌使用量；（ii）轨迹级效率，即最小化完成任务所需的总步数。基于此，我们提出DEPO——一种联合奖励简洁回应与较少动作步数的双效率偏好优化方法。在WebShop和BabyAI上的实验表明，DEPO可将令牌使用量最多降低60.9%，步骤减少26.9%，同时性能提升达29.3%。该方法还泛化至三个域外数学基准，并在仅使用25%数据训练时仍保持效率优势。项目主页：https://opencausalab.github.io/DEPO。
原文摘要 · Abstract (English)
Recent advances in large language models (LLMs) have greatly improved their reasoning and decision-making abilities when deployed as agents. Richer reasoning, however, often comes at the cost of longer chain of thought (CoT), hampering interaction efficiency in real-world scenarios. Nevertheless, there still lacks systematic definition of LLM agent efficiency, hindering targeted improvements. To this end, we introduce dual-efficiency, comprising (i) step-level efficiency, which minimizes tokens per step, and (ii) trajectory-level efficiency, which minimizes the number of steps to complete a task. Building on this definition, we propose DEPO, a dual-efficiency preference optimization method that jointly rewards succinct responses and fewer action steps. Experiments on WebShop and BabyAI show that DEPO cuts token usage by up to 60.9% and steps by up to 26.9%, while achieving up to a 29.3% improvement in performance. DEPO also generalizes to three out-of-domain math benchmarks and retains its efficiency gains when trained on only 25% of the data. Our project page is at https://opencausalab.github.io/DEPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。