让多模态大模型输出更准地反映有序标签的内在逻辑。
Latent Ordinal Evidence, Misaligned Outputs: Inference-Time Ordinal Lens Alignment for Multimodal LLMs
- 用轻量级透镜在推理时修正模型输出，增强有序信息表达。
- 在4个数据集上提升准确率16至77个百分点，最高达0.938相关性。
- 无需微调主模型，适合希望快速改进输出的开发者使用。
多模态大模型将语言模型接口用于视觉输入，而年龄估计、图像质量评估和疾病分级等有序回归任务需要对有序类别标签进行自回归决策。本文研究多模态大模型是否能可靠地将内部有序证据转化为有序数字标记输出。在四个有序基准和四个多模态大模型骨干网络上，隐藏状态中的有序标签可线性恢复，斯皮尔曼相关性高达0.938；任务设计提示进一步强化该结构。然而原生数字标记输出却弱化了这一特征：反嵌入矩阵过滤了有序方向，数字标记行空间仅保留低于1.15%的信息，且线性探测与原生输出间存在16至77个绝对点的准确率差距。为此提出**有序透镜对齐（OLA）**，一种冻结主干的推理时方法，通过在中深层解码器层训练轻量级W_S锚定透镜，融合生成有序分布，并仅修正生成时的数字标记逻辑。OLA在多数设置下优于现有最优的LoRA微调基线OrderChain，超越判别式有序基线，在所有设置下均优于离线透镜方法。
原文摘要 · Abstract (English)
Multimodal LLMs apply the language model interface to visual inputs, where ordinal regression tasks such as age estimation, image quality assessment, and disease grading require autoregressive decisions over ordered class labels. We ask whether MLLMs reliably convert internal ordinal evidence into ordered digit-token outputs. Across four ordinal benchmarks and four MLLM backbones, ordinal labels are linearly recoverable from hidden states with Spearman correlation up to 0.938, and a task-designed prompt further sharpens this structure. Yet native digit-token outputs weakly expose it: the unembedding matrix filters the ordinal direction, and the digit-token row space retains below 1.15% across all 16 model-dataset combinations, with a 16 to 77 absolute-point accuracy gap between linear-probe and native outputs. We introduce Ordinal Lens Alignment (OLA), a frozen-backbone inference-time method that trains lightweight W_S-anchored lenses on mid-to-deep decoder layers, fuses them into an ordinal distribution, and corrects only digit-token logits at generation. OLA outperforms the SOTA LoRA-tuned OrderChain baseline in most settings while keeping the MLLM frozen, surpasses discriminative ordinal baselines in most cells, and improves over an offline lens in every setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。