arXiv:2511.18415cs.MMcs.CV2025-11

提升视觉语言模型在层级决策中的连贯性,避免错误的父子关系。

DuoTeach: Dual Role Self-Teaching for Coarse-to-Fine Decision Coordination in Vision--Language Models

  • 用双重角色自教框架让模型学会跨层级一致决策
  • 在真实路径预测任务中,准确率最高提升30.24分
  • 无需标签即可训练,适合零样本泛化场景

粗粒度到细粒度的路径决策需要预测一个有效层级路径,早期决策会约束后续选择。但现有基准独立评分各层级,掩盖了跨层级有效性与一致性。为此,我们提出联合路径决策(JPD)协议,要求一次性预测完整路径,并引入深度加权前缀准确率(DWPA),通过调节权重关注深层路径可靠性。在JPD下,强视觉语言模型(VLMs)频繁出现无效的父-子节点对和脆弱的全路径预测,表明其失败不仅源于知识不全,更因跨层级决策不稳定。为此,我们提出DuoTeach——一种无需真实标签的双角色自教蒸馏框架,复用同一预训练VLM在两个角色中运行。其决策条件回溯(DCR)通过依赖先前决策生成更连贯的教师轨迹,并将这种协调行为蒸馏至学生模型,无需额外推理回溯。在多个层级结构基准及不同基础VLM上,DuoTeach在域内DWPA(alpha=0.95)最高提升30.24点,并将未见层级的零样本性能从17.17%提升至43.66%。进一步分析表明,性能提升主要归因于单次调用内多层级决策的协调能力增强。

原文摘要 · Abstract (English)

Coarse-to-fine path decision-making requires predicting a valid taxonomy path in which earlier decisions constrain later ones. However, existing benchmarks score each level independently, obscuring cross-level validity and consistency. To better align evaluation with this setting, we introduce a Joint Path Decision (JPD) protocol that requires predicting the full path in one call, together with Depth-Weighted Prefix Accuracy (DWPA), a metric family that measures path reliability with tunable emphasis on deeper levels. Under JPD, strong vision-language models (VLMs) frequently produce invalid parent-child pairs and brittle full-path predictions, suggesting that their failures stem not only from incomplete taxonomic knowledge but also from unstable cross-level decision coordination. To address this problem, we propose DuoTeach, a dual-role self-teaching distillation framework that requires no ground-truth labels and reuses the same pretrained VLM in two roles. Its Decision-Conditioned Rollout (DCR) generates more coherent teacher traces by conditioning each level on prior decisions, and distills this coordinated behavior into the student without additional test-time rollouts. Across multiple taxonomy-structured benchmarks and VLM base models, DuoTeach improves in-domain DWPA (alpha = 0.95) by up to 30.24 points and boosts zero-shot performance on unseen taxonomies from 17.17% to 43.66%. Further analyses attribute these gains to improved within-call multi-level decision coordination.

视觉语言模型层级决策自教蒸馏

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。