用轻量辅助令牌让视频编码通用化，一码多用不重训。
PAT-VCM: Plug-and-Play Auxiliary Tokens for Video Coding for Machines
- 共享压缩流+任务感知的轻量辅助令牌，无需为每任务重训练编码器。
- 在分割、深度估计等任务上表现优异，辅助令牌增益显著且开销极低。
- 适合需跨任务复用视频编码的场景，如多模态智能系统开发。
现有面向机器的视频编码（VCM）通常针对特定下游任务和模型进行训练，导致压缩表示与目标任务紧密耦合，难以扩展到多任务或适应模型更新。本文提出PAT-VCM，一种即插即用的辅助令牌框架。该框架保留共享的基线压缩流，并通过轻量级任务感知的辅助令牌进行增强，使不同下游任务可自行恢复所需信息，而无需为每个任务重新训练独立编码器。辅助信息支持三种形式：视觉残差令牌、提示/控制令牌和语义令牌。在分割、深度估计和语义识别任务上评估表明：共享检测导向的辅助分支提供可复用的首次精炼；任务特定的视觉分支提升分割与深度估计性能；提示令牌在几乎零码率成本下进一步提升分割效果；语义令牌以极低开销实现强识别性能。结果表明，共享压缩表示结合轻量任务感知辅助令牌，是替代紧耦合任务设计的一种实用且可扩展的方案。
原文摘要 · Abstract (English)
Existing video coding for machines is often trained for a specific downstream task and model. As a result, the compressed representation becomes tightly coupled to the end task, making it difficult to scale across multiple tasks or adapt to model updates. We propose PAT-VCM, a plug-and-play auxiliary-token framework for video coding for machines. PAT-VCM keeps a shared baseline compressed stream and augments it with lightweight task-aware auxiliary tokens, allowing different downstream tasks to recover the information they need without retraining a separate codec for each task. The framework supports three forms of auxiliary information: visual residual tokens, prompt/control tokens, and semantic tokens. We evaluate PAT-VCM on segmentation, depth estimation, and semantic recognition. A shared detection-oriented auxiliary branch provides a reusable first refinement, task-specific visual branches improve segmentation and depth, prompt tokens provide further segmentation gains at negligible bitrate, and semantic tokens achieve strong recognition performance with extremely low overhead. These results suggest that a shared compressed representation, combined with lightweight task-aware auxiliary tokens, is a practical and scalable alternative to tightly task-coupled VCM design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。