让视觉语言模型同时精准感知当下和回忆过去，解决长期记忆与实时推理的矛盾。
StreamTTT: Reconciling Real-Time Perception and Long-Term Memory in Streaming VLMs
- 用快速权重存储长期历史，释放注意力缓存给近期信息
- 在实时感知上比同规模模型高1.4分，长程回溯高3.7分
- 适合需要持续理解视频流的智能系统开发者
人类能同时感知当下并回忆过去，但现有流式视觉语言模型常在实时感知与长期记忆间做取舍。以往方法缩短上下文虽提升当前场景理解，却牺牲了长程记忆能力。为此，我们提出StreamTTT，将长程历史写入外部的在线更新快速权重，避免占据注意力上下文；同时保留短滑动键值缓存用于最新证据，缓解注意力稀释。我们在离线长视频问答与新构建的实时问答数据集上联合训练。在OVO-Bench上，按各自报告输入协议，StreamTTT-4B在实时感知上优于同规模SimpleStream-4B 1.4分，在后向追踪任务上领先3.7分；在StreamingBench的实时视觉理解子集上，性能媲美更大规模的SimpleStream-8B。代码将公开。
原文摘要 · Abstract (English)
Humans effortlessly perceive the present while remembering the past, yet streaming VLMs often trade off real-time perception against long-term memory. Prior work shows that shortening the context can sharpen current-scene perception at the expense of long-range recall. To reconcile these abilities, we introduce StreamTTT, which writes long-range history into online-updated fast weights outside the attention context. This leaves a short sliding key-value cache dedicated to recent evidence, mitigating attention dilution. We train StreamTTT jointly on offline long-video QA and a newly constructed real-time QA corpus. On OVO-Bench, under each model's reported input protocol, StreamTTT-4B outperforms the same-scale SimpleStream-4B by 1.4 points in real-time perception and 3.7 points in backward tracing. It also remains competitive with the larger SimpleStream-8B on StreamingBench's Real-Time Visual Understanding (RTVU) subset. Our code will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。