用预测误差决定视频分块,让模型能长期准确预测。
SUNTA: Hierarchical Video Prediction with Surprise-based Chunking

- 根据预测误差动态划分时间片段,更贴合数据内在结构
- 在250步长的视频预测中保持高精度,基线模型10步内就崩溃
- 适合需要长时序建模的视频生成与强化学习任务
层级状态空间模型(HSSMs)通过将序列分割为时间片段,实现长时程预测。然而其性能依赖于片段边界的设定。以往方法多采用固定长度或基于相似性的边界检测,常与数据内在时间结构不符。本文主张以预测误差(即‘意外’)驱动分块,更直接反映何时需引入长程上下文。但将意外机制融入HSSMs面临两大挑战:端到端训练中的层级坍塌,以及开环预测时缺乏意外信号。为此,我们提出惊喜式嵌套时间抽象(SUNTA),采用解耦训练策略保留意外信号,并利用内部不一致性作为自上而下的意外度量,确定想象回溯中的片段边界。在二维和三维环境的视频预测任务中,SUNTA表现优于基线,独特地在250个时间步内维持准确预测,而所有基线均在前10步内显著退化。
原文摘要 · Abstract (English)
Hierarchical state-space models (HSSMs) offer a promising approach to long-horizon prediction by segmenting sequences into temporal chunks. However, their performance hinges on how chunk boundaries are determined. While prior HSSMs typically rely on fixed-length chunking or similarity-based boundary detection, these methods often misalign with the intrinsic temporal structure of the data. We argue that chunking should instead be driven by prediction errors, which more directly indicate when longer-range context becomes necessary. Nevertheless, integrating surprise-based chunking into HSSMs introduces critical challenges, including hierarchical collapse during end-to-end training and the absence of surprise signals during open-loop prediction. To address these issues, we propose Surprise-based Nested Temporal Abstraction (SUNTA), a method that employs a decoupled training strategy to preserve surprise signals and uses internal inconsistency as a top-down surprise metric to determine chunk boundaries within imagined rollouts. Experiments on video prediction tasks in 2D and 3D environments demonstrate that SUNTA outperforms baselines, uniquely maintaining accurate predictions over 250 timesteps, whereas all baselines degrade within the first 10 timesteps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。