首次建立多模态检索的信息论极限，给出高效编码的设计准则。
Rate-Distortion Limits for Multimodal Retrieval: Theory, Optimal Codes, and Finite-Sample Guarantees
- 将排序问题建模为有损编码，推导出可计算的速率-失真函数。
- 自适应温度量化器在30k数据集上逼近理论极限，误差小于2个百分点。
- 适用于设计熵感知对比学习、持续学习检索器等系统。
我们首次建立了多模态检索的信息论极限。将排序视为有损源编码，推导出互斥排名失真下的单字母速率-失真函数 $R(D)$，并证明其逆向界由模态均衡项与刻画熵不平衡和跨模态冗余的偏斜惩罚 $κ riangle H$ 组成。随后构建了一种显式的熵加权随机量化器，配合自适应的每模态温度解码器；通过Blahut-Arimoto论证，该方案在 $n$ 个训练三元组下，失真距 $R(D)$ 仅差 $O(n^{-1})$。VC型分析得出首个有限样本超额风险界，其复杂度在模态数和熵差上均呈次线性增长。在控制高斯混合与 Flickr30k 上的实验表明，我们的自适应编码接近理论前沿，误差不超过两个百分点，而固定温度与朴素CLIP基线则显著落后。结果共同给出了高质量多模态检索所需比特数的理论答案，并为熵感知对比目标、持续学习检索器及检索增强生成器提供设计指导。
原文摘要 · Abstract (English)
We establish the first information-theoretic limits for multimodal retrieval. Casting ranking as lossy source coding, we derive a single-letter rate-distortion function $R(D)$ for reciprocal-rank distortion and prove a converse bound that splits into a modality-balanced term plus a skew penalty $κ\,ΔH$ capturing entropy imbalance and cross-modal redundancy. We then construct an explicit entropy-weighted stochastic quantizer with an adaptive, per-modality temperature decoder; a Blahut-Arimoto argument shows this scheme achieves distortion within $O(n^{-1})$ of $R(D)$ using $n$ training triples. A VC-type analysis yields the first finite-sample excess-risk bound whose complexity scales sub-linearly in both the number of modalities and the entropy gap. Experiments on controlled Gaussian mixtures and Flickr30k confirm that our adaptive codes sit within two percentage points of the theoretical frontier, while fixed-temperature and naive CLIP baselines lag significantly. Taken together, our results give a principled answer to "how many bits per query are necessary" for high-quality multimodal retrieval and provide design guidance for entropy-aware contrastive objectives, continual-learning retrievers, and retrieval-augmented generators.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。