让视频大模型轻松接入新模态，无需重新训练
V-LynX: Token Interface Alignment for Video+X LLMs
- 用轻量辅助路径对齐视觉与新模态的注意力和分布
- 在多个任务上达到当前最优效果，且效率高
- 适合想快速扩展视频模型能力的研究者
本研究揭示了视频大模型的一个关键现象：视觉标记并非简单转换为文本嵌入，而是形成连续的流形接口，使视觉标记可在模型内作为独立实体运行。基于此发现，我们提出V-LynX框架，通过复用内部接口实现多模态的可扩展集成。与传统需依赖专用编码器或成对监督的方法不同，V-LynX采用轻量级并行辅助路径，配合冻结的视觉编码器，利用无配对单模态数据集对齐注意力响应与统计分布，确保流形兼容性的同时保持视频大模型完整性。大量基准测试显示，V-LynX在音视频问答、3D推理、高帧率视频及多视角理解任务中均达最优性能，兼具高效性。代码已开源。
原文摘要 · Abstract (English)
This study introduces an intriguing phenomenon in Video LLMs: rather than merely translating frames into textual embeddings, Video LLMs establish a continuous manifold, token interface, allowing visual tokens to operate as standalone entities within the architecture. Exploiting this discovery, we propose V-LynX, a scalable framework that integrates novel modalities into Video LLMs by repurposing the internalized interface. Departing from conventional paradigms that necessitate heavy modality-specific encoders or paired supervision, V-LynX employs a lightweight auxiliary pathway in parallel with the frozen vision encoder. Our method integrates new sensory inputs with intrinsic video priors by aligning both attention responses and statistical distributions using unpaired unimodal data sets. This ensures manifold compatibility while preserving the integrity of the Video LLMs. Extensive benchmarks demonstrate that V-LynX achieves SOTA and efficiency across audio-visual QA, 3D reasoning, high-frame-rate, and multi-view video understanding. The code is available at https://github.com/park-jungin/lynx.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。