一个模型搞定文档检索与生成，节省一半显存。
Hydra: Unifying Document Retrieval and Generation in a Single Vision-Language Model
- 单视觉语言模型双头设计，用LoRA切换检索或生成模式。
- 4B模型显存从28.85GB降至10.77GB，降幅62.7%。
- 支持零样本音频检索，适合多模态系统部署者。
视觉文档理解通常需独立的检索与生成模型，导致内存和系统复杂度翻倍。我们提出Hydra，一种双头方法，可在单一视觉语言模型中实现类似ColBERT的后期交互式检索与自回归生成。推理时仅通过开启或关闭一个专用的LoRA适配器即可切换功能：启用时生成多向量嵌入用于检索；禁用时恢复基模型生成质量，其426个语言模型权重张量与全新加载的Qwen3.5-4B完全一致（逐字节相同）。我们识别出两类会无声破坏生成能力的故障模式（注意力模式恢复与lm_head保留），以及一项效率要求（KV缓存感知解码）；Hydra在结构上规避前两者，并在解码循环中处理第三项。我们发布了两个规模版本：Hydra-4B与Hydra-0.8B，共享LoRA超参数（r=32, alpha=32）及优化方案，仅数据混合与投影维度不同。单模型设计使4B模型峰值显存从28.85GB降至10.77GB（减少62.7%），0.8B模型则从5.79GB降至2.37GB（减少59.1%），相比共驻留双模型部署显著降低。受控消融实验表明，GritLM式的联合训练在检索性能上等同于Hydra的仅检索训练，但其在生成模式下使用LoRA时性能崩溃。以Qwen2.5-Omni-3B为基底的原型验证了生成一致性可迁移至非Qwen3.5骨干网络，并在图像检索上达到比Hydra-4B低2-8个百分点的性能；同时，通过冻结Whisper编码器实现了零样本音频检索。
原文摘要 · Abstract (English)
Visual document understanding typically requires separate retrieval and generation models, doubling memory and system complexity. We present Hydra, a dual-head approach that provides both ColBERT-style late-interaction retrieval and autoregressive generation from a single vision-language model. A single LoRA adapter, trained only for retrieval, is toggled at inference: enabling it produces multi-vector embeddings; disabling it recovers the base model's generation quality, with 426 of 426 language-model weight tensors byte-for-byte identical to a freshly-loaded Qwen3.5-4B. We identify two failure modes that can silently break generation in retrieval-fine-tuned VLMs (attention-mode restoration and lm_head preservation) plus an efficiency requirement (KV-cache-aware decoding); Hydra sidesteps the first two structurally and addresses the third in the decode loop. We release two scales, Hydra-4B and Hydra-0.8B, sharing LoRA hyperparameters (r=32, alpha=32) and optimisation recipe; data mix and projection dim differ across scales. The single-model design cuts peak GPU memory from 28.85 GB to 10.77 GB at 4B (62.7% reduction) and from 5.79 GB to 2.37 GB at 0.8B (59.1%) relative to a co-resident two-model deployment. A controlled ablation finds GritLM-style joint training matches Hydra's retrieval-only training on the evaluated modes while its LoRA-on generation mode collapses. A proof-of-concept on Qwen2.5-Omni-3B preserves generation equivalence on a non-Qwen3.5 backbone and transfers image retrieval within 2-8 pp of Hydra-4B, with zero-shot audio retrieval emerging through the frozen Whisper encoder.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。