用分阶段策略让弱Q网络先达标再优化，提升强化学习稳定性和效率。
Sat-EnQ: Satisficing Ensembles of Weak Q-Learners for Reliable and Compute-Efficient Reinforcement Learning
- 先学‘够好’再冲刺：用动态基线控制早期值估计，防过拟合
- 实测方差降3.8倍，失败率从50%降至0%，噪声环境下仍保79%性能
- 计算量仅传统集成的40%，适合资源有限但需可靠的RL应用
深度Q学习算法在训练初期因最大化操作放大估计误差而极不稳定。受有限理性理论与发育学习启发，我们提出双阶段框架Sat-EnQ：第一阶段以满足性目标训练一组轻量级Q网络，通过动态基线限制早期值增长，生成多样且低方差的估计，避免灾难性过估计；第二阶段将集成蒸馏为大型网络，并用标准Double DQN微调。理论上证明满足性可诱导有界更新，不会增加目标方差，且可量化大幅降低条件。实验表明，Sat-EnQ实现3.8倍方差降低，消除灾难性失败（0%对DQN的50%），在环境噪声下保持79%性能，计算量仅为引导集成的2.5倍。结果揭示了先满足后优化的可靠强化学习路径。
原文摘要 · Abstract (English)
Deep Q-learning algorithms remain notoriously unstable, especially during early training when the maximization operator amplifies estimation errors. Inspired by bounded rationality theory and developmental learning, we introduce Sat-EnQ, a two-phase framework that first learns to be ``good enough'' before optimizing aggressively. In Phase 1, we train an ensemble of lightweight Q-networks under a satisficing objective that limits early value growth using a dynamic baseline, producing diverse, low-variance estimates while avoiding catastrophic overestimation. In Phase 2, the ensemble is distilled into a larger network and fine-tuned with standard Double DQN. We prove theoretically that satisficing induces bounded updates and cannot increase target variance, with a corollary quantifying conditions for substantial reduction. Empirically, Sat-EnQ achieves 3.8x variance reduction, eliminates catastrophic failures (0% vs 50% for DQN), maintains 79% performance under environmental noise}, and requires 2.5x less compute than bootstrapped ensembles. Our results highlight a principled path toward robust reinforcement learning by embracing satisficing before optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。