Aftab提升无缓存Q网络性能，通过优化编码器与价值函数设计。
Aftab: A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks
- 设计新型并行化Q网络编码器结构，融合多重特征交互机制。
- 在Atari-57上实现6.592的IQM人类归一化得分，优于基准模型2.715。
- 适合追求高效、低内存消耗强化学习框架的研究者使用。
近期深度强化学习趋向于简化且高度并行化的范式。并行化Q网络（PQN）算法可在无需经验回放缓冲区或目标网络的情况下实现离策略值学习。然而，在此类无缓冲设置中，视觉编码器的表征能力与计算效率仍缺乏系统研究。本文系统考察了PQN中卷积神经网络的架构设计空间，评估了八种不同拓扑结构，并明确刻画其参数量与计算开销。进一步通过引入Hadamax编码范式，结合分类、集成与双重价值头，研究乘法型表示学习与先进价值估计的影响。在Atari-57上的大量实验表明，最终复合架构Aftab达到6.592的四分位均值（IQM）人类归一化得分，相较标准PQN基线2.715显著提升，同时具有0.86的改进概率。在Procgen-Hard上，Aftab的归一化学习曲线曲线下面积（nAUC）达0.541，优于PQN的0.216。结果表明，精心设计的编码器拓扑、乘法特征交互及先进价值估计头可显著提升无缓存并行化Q学习的性能，同时保持其内存高效训练特性。完整Aftab框架（含模型定义、训练配置、复现设置与原始日志）已开源：https://github.com/tahashieenavaz/aftab
原文摘要 · Abstract (English)
Recent advancements in deep reinforcement learning have increasingly favored simplified, highly parallelized paradigms. Notably, the Parallelized Q-Network (PQN) algorithm enables off-policy value learning without relying on experience replay buffers or target networks. However, the representational capacity and computational efficiency of visual encoders operating in these buffer-free settings remain comparatively underexplored. In this work, we systematically investigate the architectural design space of Convolutional Neural Networks within PQN. We evaluate eight distinct CNN topologies while explicitly characterizing their parameter and computational requirements. We further study the effect of multiplicative representation learning and advanced value estimation by integrating the Hadamax encoding paradigm with categorical, ensemble, and dueling value heads. Extensive experiments on Atari-57 show that our final composite architecture, Aftab, achieves an Interquartile Mean (IQM) Human-Normalized Score of 6.592, compared with 2.715 for the standard PQN baseline, together with a 0.86 Probability of Improvement over PQN. We additionally evaluate Aftab on Procgen-Hard to assess performance under procedurally varying visual environments. Aftab achieves a normalized learning-curve Area Under the Curve (nAUC) of 0.541 compared with 0.216 for PQN. Overall, the results demonstrate that carefully designed encoder topology, multiplicative feature interactions, and advanced value-estimation heads can substantially improve performance within a parallelized, replay-free Q-learning framework while preserving its memory-efficient training paradigm. The complete Aftab framework, including model definitions, training configurations, reproducibility settings, and raw experimental logs, is open-sourced at https://github.com/tahashieenavaz/aftab
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。