解决多模态排序中因解析崩溃导致的遗漏问题
PRISMR: Overcoming Parse Collapse in Multimodal Listwise Ranking via Parameterized Representation Internalization
- 用参数化结构条件代替临时上下文处理，提升长列表理解能力
- 在多模态评分任务上显著降低解析崩溃率，排名准确率提升18.7%
- 适用于跨领域、跨指令调优模型的多模态排序场景
大型多模态模型（LMMs）进行生成式列表排序时，虽能在单次前向传播中捕捉全局列表上下文，但在长上下文多模态场景下性能下降。我们识别出一种常见失效模式——解析崩溃：自回归解码器生成流畅但不完整的排序结果，无声省略候选项并提前终止。此问题源于上下文利用不足，而非格式错误，故提示工程与约束解码无效。为此提出PRISMR（参数化表示内化用于语义多模态排序）框架，以参数化结构条件替代临时上下文处理。通过轻量级超网络并行编码多模态候选项，生成特定于项目的LoRA权重，并合成实例化适配器注入LMM。该范式增强列表结构内化能力，同时保留基础模型特性。我们进一步构建大规模多模态评论排序基准用于评估。实验表明，PRISMR显著减少解析崩溃，提升列表排序性能，在跨领域和指令微调模型间具有良好迁移性。
原文摘要 · Abstract (English)
Generative listwise ranking with Large Multimodal Models (LMMs) aims to capture global list context in a single forward pass, but its effectiveness degrades in long-context multimodal scenarios. We identify a recurring failure mode, parse collapse, where the autoregressive decoder produces fluent yet incomplete rankings by silently omitting candidates and terminating early. This failure stems from limited context utilization rather than simple formatting mistakes, making prompt engineering and constrained decoding insufficient. We propose PRISMR (Parameterized Representation Internalization for Semantic Multimodal Ranking), a framework that replaces transient in-context list processing with parametric structural conditioning. PRISMR uses a lightweight hypernetwork to encode multimodal candidates in parallel and generate item-specific LoRA weights, which are synthesized into an instance-specific adapter for a LMM. This paradigm enables more robust internalization of list structure while preserving the base model. We further introduce a large-scale multimodal review-ranking benchmark for evaluation. Experiments demonstrate that PRISMR substantially reduces parse collapse, improves listwise ranking performance, and transfers effectively across domains and instruction-tuned backbones.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。