通过在线学习提升流式强化学习的表征能力，让每帧数据都更有价值。
Squeezing More from the Stream : Learning Representation Online for Streaming Reinforcement Learning
- 引入自预测表征机制，从瞬时数据中提取有效信息。
- 在Atari等3个基准上超越现有流式方法，性能接近有回放缓冲的模型。
- 适合资源受限设备上的实时强化学习应用。
在流式强化学习中，状态转移数据在单次更新后即被丢弃，虽节省资源但导致智能体样本效率极低，因仅靠价值函数损失难以从短暂数据中提取有意义表征。本文将自预测表征（SPR）扩展至流式框架，最大化每帧数据的利用价值。然而，流式场景下样本高度相关，直接应用辅助损失会导致训练不稳定。为此，我们提出相对于动量目标的正交梯度更新，解决流式优化器引发的梯度冲突问题。在Atari、MinAtar和Octax系列任务上验证，该方法系统性优于现有流式基线。潜空间分析（包括t-SNE可视化与有效秩测量）表明，本方法学习到的表征显著更丰富，弥合了无回放缓冲带来的性能差距，同时仅需少量CPU核心即可高效训练。
原文摘要 · Abstract (English)
In streaming Reinforcement Learning (RL), transitions are observed and discarded immediately after a single update. While this minimizes resource usage for on-device applications, it makes agents notoriously sample-inefficient, since value-based losses alone struggle to extract meaningful representations from transient data. We propose extending Self-Predictive Representations (SPR) to the streaming pipeline to maximize the utility of every observed frame. However, due to the highly correlated samples induced by the streaming regime, naively applying this auxiliary loss results in training instabilities. Thus, we introduce orthogonal gradient updates relative to the momentum target and resolve gradient conflicts arising from streaming-specific optimizers. Validated across the Atari, MinAtar, and Octax suites, our approach systematically outperforms existing streaming baselines. Latent-space analysis, including t-SNE visualizations and effective-rank measurements, confirms that our method learns significantly richer representations, bridging the performance gap caused by the absence of a replay buffer, while remaining efficient enough to train on just a few CPU cores.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。