多向量检索模型在长文本查询上严重失效，且调优反致性能下降。
Reproduction Beyond Benchmarks: ConstBERT and ColBERT-v2 Across Backends and Query Distributions
- 通过五维评估发现模型对长叙事查询鲁棒性差
- 长查询下MRR@10下降86%-97%，20词后性能饱和
- 适合关注模型真实泛化能力的研究者
可复现性应验证架构鲁棒性，而不仅是数值准确。我们对ColBERT-v2和ConstBERT进行五维度评估，发现尽管ConstBERT在MS-MARCO上能保持0.05%的MRR@10误差，但两者在长篇叙事查询（TREC ToT 2025）中均出现86%-97%的性能下降。消融实验表明，此失败源于架构缺陷：MaxSim算子采用统一词元权重，无法区分有效信号与冗余噪声，导致性能在20词后趋于饱和。此外，未公开的后端参数造成8分差距，而使用3倍数据微调反而使性能最多恶化29%。结论是，多向量检索的架构限制无法仅靠适应策略突破。代码：https://github.com/utshabkg/multi-vector-reproducibility。
原文摘要 · Abstract (English)
Reproducibility must validate architectural robustness, not just numerical accuracy. We evaluate ColBERT-v2 and ConstBERT across five dimensions, finding that while ConstBERT reproduces within 0.05% MRR@10 on MS-MARCO, both models show a drop of 86-97% on long, narrative queries (TREC ToT 2025). Ablations prove this failure is architectural: performance plateaus at 20 words because the MaxSim operator's uniform token weighting cannot distinguish signal from filler noise. Furthermore, undocumented backend parameters create an 8-point gap due to ConstBERT's sparse centroid coverage, and fine-tuning with 3x more data actually degrades performance by up to 29%. We conclude that architectural constraints in multi-vector retrieval cannot be overcome by adaptation alone. Code: https://github.com/utshabkg/multi-vector-reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。