通过时间抽象对齐前后向表示的谱结构,提升连续控制中的长时序表征能力。
Spectral Alignment in Forward-Backward Representations via Temporal Abstraction
- 利用时间抽象作为低通滤波器,抑制高频率谱分量以降低有效秩。
- 在高折扣因子下显著提升学习稳定性,同时保持值函数误差的理论边界。
- 适用于需要长时序建模的连续控制任务,尤其适合高精度值函数估计场景。
前后向(FB)表示通过低秩分解提供连续空间中成功表示(SR)的学习框架,但连续环境的高秩转移动态与FB架构的低秩瓶颈之间常存在根本性的谱失配,导致低秩表示学习困难。本文分析时间抽象作为一种缓解该失配的机制:通过刻画转移算子的谱特性,发现时间抽象类似低通滤波器,能抑制高频谱成分,从而降低诱导出的SR的有效秩,同时保持值函数误差的严格上界。实验证明,这种谱对齐是稳定FB学习的关键,尤其在高折扣因子下,此时自举法易引发误差累积。结果表明,时间抽象是一种有原则的机制,可调控马尔可夫决策过程的谱结构,实现连续控制中的高效长时序表示。
原文摘要 · Abstract (English)
Forward-backward (FB) representations provide a powerful framework for learning the successor representation (SR) in continuous spaces by enforcing a low-rank factorization. However, a fundamental spectral mismatch often exists between the high-rank transition dynamics of continuous environments and the low-rank bottleneck of the FB architecture, making accurate low-rank representation learning difficult. In this work, we analyze temporal abstraction as a mechanism to mitigate this mismatch. By characterizing the spectral properties of the transition operator, we show that temporal abstraction acts analogously to a low-pass filter that suppresses high-frequency spectral components. This suppression reduces the effective rank of the induced SR while preserving a formal bound on the resulting value function error. Empirically, we show that this alignment is a key factor for stable FB learning, particularly at high discount factors where bootstrapping becomes error-prone. Our results identify temporal abstraction as a principled mechanism for shaping the spectral structure of the underlying MDP and enabling effective long-horizon representations in continuous control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。