针对移动端导航强化学习样本效率低的问题，提出区分正负样本的更新策略。
Succeed or Learn Slowly: Sample Efficient Off-Policy Reinforcement Learning for Mobile App Control
- 正样本直接更新，负样本保守正则化，防止模型退化
- 在AndroidWorld上相对提升至少17%，推理速度比GPT-4o快5-60倍
- 适合资源受限环境下基于大模型的移动端智能控制
基于基础模型进行多轮任务策略近似时，强化学习仍面临挑战。本文指出稀疏奖励与策略梯度更新是两大瓶颈，并提出核心洞察：高回报正样本更新无需正则化，而负样本反映不良行为，过度更新会损害模型性能。为此提出新算法Succeed or Learn Slowly（SoLS），采用改进的离线策略演员-评论家框架，在移动端应用控制任务中实现高效微调。该方法对正样本实施直接策略更新，对负样本采用保守正则化更新，避免模型退化。同时引入成功轨迹重放（STR），优先学习成功交互，进一步提升样本效率。在AndroidWorld基准测试中，SoLS显著优于现有方法（相对提升至少17%），包括提示工程与强化学习方案；且计算资源消耗远低于GPT-4o基方法，推理速度提升5至60倍。
原文摘要 · Abstract (English)
Reinforcement learning (RL) using foundation models for policy approximations in multi-turn tasks remains challenging. We identify two main limitations related to sparse reward settings and policy gradient updates, based on which we formulate a key insight: updates from positive samples with high returns typically do not require policy regularisation, whereas updates from negative samples, reflecting undesirable behaviour, can harm model performance. This paper introduces Succeed or Learn Slowly (SoLS), a novel off-policy RL algorithm evaluated on mobile app control tasks. SoLS improves sample efficiency when fine-tuning foundation models for user interface navigation via a modified off-policy actor-critic approach, applying direct policy updates for positive samples and conservative, regularised updates for negative ones to prevent model degradation. We augment SoLS with Successful Transition Replay (STR), which prioritises learning from successful interactions, further improving sample efficiency. We evaluate SoLS on the AndroidWorld benchmark, where it significantly outperforms existing methods (at least 17% relative increase), including prompt-engineering and RL approaches, while requiring substantially fewer computational resources than GPT-4o-based methods with 5-60x faster inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。