用动态生成的提示词实现视频问答持续学习，减少遗忘且不增加内存。
HyperTokens: Controlling Token Dynamics for Continual Video-Language Understanding
- 基于Transformer的提示词生成器，按需生成微调令牌，固定内存使用。
- 在两个标准基准上平均准确率更高，遗忘率显著降低。
- 适合需要持续学习多模态模型的研究者和应用开发者。
多模态大模型进行持续视频问答时，面临任务间干扰和存储特定任务提示的高昂成本问题。我们提出HyperTokens，一种基于Transformer的令牌生成器，可按需生成微调令牌，在保持内存不变的同时实现对提示更新的显式控制。为抑制遗忘，我们设计了受元学习启发的正则化项，通过前瞻避免任务特异性尖锐方向，并将演化中的生成器锚定于先前任务。进一步地，我们将目标与尖锐感知优化相联系，揭示其促进更平坦跨任务最小值并提升保留能力的原因。除正则化外，HyperTokens还通过共享生成权重引入轻量级辅助多模态监督；基于因果视角，设计可行的目标函数与代理互信息损失，以正则化反因果跨模态方向。在两个标准持续视频问答基准上，HyperTokens实现了更高的平均准确率且显著降低遗忘。最后，我们引入一个具有挑战性的跨模态图像问答→视频问答迁移协议，证明HyperTokens在此设置下仍能实现稳健的持续迁移。
原文摘要 · Abstract (English)
Continual VideoQA with multimodal LLMs is hindered by interference between tasks and the prohibitive cost of storing task-specific prompts. We introduce HyperTokens, a transformer-based token generator that produces fine-tuning tokens on demand, giving explicit control over prompt updates while keeping memory fixed. To suppress forgetting, we propose meta-inspired regularisers that look ahead to avoid task-specific sharp directions and anchor the evolving generator to prior tasks. We further connect our objective to sharpness-aware optimisation, providing insight into why it encourages flatter cross-task minima and improves retention. Beyond regularisation, HyperTokens exploits lightweight auxiliary multimodal supervision through shared generation weights; guided by a causal perspective, we design feasible objectives and surrogate mutual-information losses to regularise anti-causal cross-modal directions. Across two standard continual VideoQA benchmarks, HyperTokens achieves higher average accuracy with substantially lower forgetting. Finally, we introduce a challenging cross-modal ImageQA->VideoQA protocol and show that HyperTokens enables robust continual transfer in this setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。