arXiv:2602.11805cs.LG2026-02

将路径签名分解为时间有序的增量贡献,提升强化学习中的实时响应能力

From Path Signatures to Sequential Modeling: Incremental Signature Contributions for Offline RL

  • 将路径签名拆解为按时间顺序的增量成分,保留时序动态性
  • 在多个机器人控制任务中显著优于传统方法,尤其在延迟奖励和低质量数据下表现稳健
  • 适合需要高灵敏度与稳定性的实时控制场景,如机器人运动规划

路径签名可将轨迹嵌入张量代数,构成路径的通用非参数表示;然而标准形式将时序结构压缩为单一全局对象,限制了其在需逐步反应决策任务中的应用。本文提出增量签名贡献(ISC)方法,将截断路径签名分解为张量代数空间中时间有序的元素序列,对应路径最后增量引发的增量贡献。该重构保持了签名的代数结构与表达能力,同时显式揭示其内部时序演化,使基于签名的表示可被序列建模方法处理。相比完整签名,ISC天然对轨迹瞬时更新敏感,这对敏感且需稳定性的控制动态至关重要。基于此表示,我们提出ISC-Transformer(ISCT),一种将ISC集成到标准Transformer架构的离线强化学习模型,无需额外结构修改。我们在HalfCheetah、Walker2d、Hopper和Maze2d上评估ISCT,涵盖延迟奖励和降级数据集设置。结果表明,ISC为时序敏感控制任务提供了理论坚实且实践有效的路径处理替代方案。

原文摘要 · Abstract (English)

Path signatures embed trajectories into tensor algebra and constitute a universal, non-parametric representation of paths; however, in the standard form, they collapse temporal structure into a single global object, which limits their suitability for decision-making problems that require step-wise reactivity. We propose the Incremental Signature Contribution (ISC) method, which decomposes truncated path signatures into a temporally ordered sequence of elements in the tensor-algebra space, corresponding to incremental contributions induced by last path increments. This reconstruction preserves the algebraic structure and expressivity of signatures, while making their internal temporal evolution explicit, enabling processing signature-based representations via sequential modeling approaches. In contrast to full signatures, ISC is inherently sensitive to instantaneous trajectory updates, which is critical for sensitive and stability-requiring control dynamics. Building on this representation, we introduce ISC-Transformer (ISCT), an offline reinforcement learning model that integrates ISC into a standard Transformer architecture without further architectural modification. We evaluate ISCT on HalfCheetah, Walker2d, Hopper, and Maze2d, including settings with delayed rewards and downgraded datasets. The results demonstrate that ISC method provides a theoretically grounded and practically effective alternative to path processing for temporally sensitive control tasks.

强化学习路径签名序列建模离线学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。