小模型用GRPO控制无人机时，连续动作空间会导致失效，离散动作可成功收敛。
Discrete Action Space as a Prerequisite for GRPO Convergence in Small-Model Continuous Control
- 将连续动作改为5类PID预设的离散选择，使训练稳定收敛
- 最高达98.6%成功率，64步时抖动仅0.656 m/s³
- 适用于小模型强化学习，尤其适合对稳定性要求高的控制任务
我们研究了组相对策略优化（GRPO）能否微调小型语言模型完成模拟四旋翼速度控制任务。在基准测试中，对Qwen-0.5B使用原始GRPO进行25 Hz四旋翼速度控制微调，在60步内成功率降至0%，熵值从0.35下降至0.03，陷入零动作崩溃。两个消融实验——移除加速度惩罚项或移除对预训练先验的KL锚定——均能防止熵崩溃，但均无法实现学习。当动作接口改为5类PID参数预设的离散选择后，训练成功收敛。所得控制器在平滑性与可靠性间形成帕累托前沿：64步时达到98.6%成功率、0.656 m/s³抖动；256步时实现100%成功率、1.103 m/s³抖动（或在匹配速度限制下为0.796）。该方法在三个预训练语言模型上均验证有效。作为对照，经重新调优的经典基线PID（Ki=0.30，vmax=2.5）在0.736 m/s³抖动下也达到100%成功率。高保真仿真使用Crazyflie 2.1动力学揭示了悬停区域的训练分布差距。
原文摘要 · Abstract (English)
We study whether Group Relative Policy Optimization (GRPO) can fine-tune small language models for simulated quadrotor continuous-control tasks. In our benchmark, vanilla GRPO fine-tuning of Qwen-0.5B for 25 Hz quadrotor velocity control collapses to the trivial zero action: 0 percent success rate, with entropy falling from 0.35 to 0.03 within 60 steps. Two ablations - removing the jerk-penalty term and removing the KL anchor to the pretrained prior - each prevent entropy collapse, yet neither enables learning. When the action interface is replaced by a 5-way categorical choice over PID presets, training converges. The resulting controller traces a smoothness-reliability Pareto frontier along training duration; both endpoints are reported: 98.6 percent success with 0.656 m/s3 jerk at 64 steps, and 100 percent success with 1.103 m/s3 jerk, or 0.796 under a matched velocity cap, at 256 steps. The recipe is evaluated across three pretrained language models. As context, a re-tuned classical baseline, PID with Ki = 0.30 and vmax = 2.5, reaches the same 100 percent success rate at jerk 0.736 m/s3. A high-fidelity simulation using Crazyflie 2.1 dynamics surfaces a hover-region training-distribution gap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。