GIPO通过软化重要性采样权重，提升强化学习在稀疏数据下的训练效率与稳定性。
GIPO: Gaussian Importance Sampling Policy Optimization
- 用高斯信任权重替代硬截断，平滑极端重要性比率
- 在不同回放缓冲区大小下均达领先性能，样本效率显著提升
- 适合数据稀缺或过时场景，尤其适用于多模态智能体的后训练
后训练阶段的强化学习近年来在推动多模态智能体超越监督模仿方面展现出强大潜力。然而，强化学习仍受限于数据效率低下，特别是在交互数据稀缺且迅速过时的场景中。为此，本文提出GIPO（Gaussian Importance Sampling Policy Optimization），一种基于截断重要性采样的策略优化目标，将硬截断替换为基于对数比率的高斯信任权重，以软性抑制极端重要性比率，同时保持非零梯度。理论分析表明，GIPO引入了隐式的、可调的更新幅度约束，而集中界保证了在有限样本估计下的鲁棒性与稳定性。实验结果表明，GIPO在从近在线到高度过时数据的广泛回放缓冲区大小下，均优于现有截断基线方法，表现出更优的偏差-方差权衡、更高的训练稳定性以及改进的样本效率。代码已开源：https://github.com/distanceLu/GIPO。
原文摘要 · Abstract (English)
Post-training with reinforcement learning (RL) has recently shown strong promise for advancing multimodal agents beyond supervised imitation. However, RL remains limited by poor data efficiency, particularly in settings where interaction data are scarce and quickly become outdated. To address this challenge, GIPO (Gaussian Importance sampling Policy Optimization) is proposed as a policy optimization objective based on truncated importance sampling, replacing hard clipping with a log-ratio-based Gaussian trust weight to softly damp extreme importance ratios while maintaining non-zero gradients. Theoretical analysis shows that GIPO introduces an implicit, tunable constraint on the update magnitude, while concentration bounds guarantee robustness and stability under finite-sample estimation. Experimental results show that GIPO achieves state-of-the-art performance among clipping-based baselines across a wide range of replay buffer sizes, from near on-policy to highly stale data, while exhibiting superior bias--variance trade-off, high training stability and improved sample efficiency. Code is available at https://github.com/distanceLu/GIPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。