通过自生成知识提示提升视觉语言模型在农业领域的准确率
Look, Recite, Then Answer: Enhancing VLM Performance via Self-Generated Knowledge Hints
- 三阶段框架：看、背、答，主动激活模型内部知识
- 农业数据集上比Qwen2-VL-72B高23.52%准确率
- 无需外部搜索，适合需要可靠推理的垂直领域
视觉语言模型在精准农业等专业领域表现受限，主要因‘推理驱动幻觉’——语言先验覆盖视觉感知。核心瓶颈是‘模态鸿沟’：视觉嵌入无法有效触发模型参数中已编码的细粒度专家知识。我们提出‘看、背、答’框架，以参数高效方式增强视觉语言模型，保持主干模型冻结。该框架将推理分为三步：(1) 看阶段生成客观视觉描述与候选集；(2) 背阶段使用轻量级1.7B路由器，将视觉线索转化为目标查询，触发特定候选的知识；(3) 答阶段并行对齐描述与复述知识，选择最一致标签。在AgroBench上，本方法达到当前最优性能，相比Qwen2-VL-72B Weed Identification准确率提升23.52%，且超越GPT-4o，无外部搜索开销。模块化设计通过将被动感知转为可控知识检索，缓解幻觉问题。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) exhibit significant performance plateaus in specialized domains like precision agriculture, primarily due to "Reasoning-Driven Hallucination" where linguistic priors override visual perception. A key bottleneck is the "Modality Gap": visual embeddings fail to reliably activate the fine-grained expert knowledge already encoded in model parameters. We propose "Look, Recite, Then Answer," a parameter-efficient framework that enhances VLMs via self-generated knowledge hints while keeping backbone models frozen. The framework decouples inference into three stages: (1) Look generates objective visual descriptions and candidate sets; (2) Recite employs a lightweight 1.7B router to transform visual cues into targeted queries that trigger candidate-specific parametric knowledge; (3) Answer performs parallel evidence alignment between descriptions and recited knowledge to select the most consistent label. On AgroBench, our method achieves state-of-the-art results, improving Weed Identification accuracy by 23.52% over Qwen2-VL-72B and surpassing GPT-4o without external search overhead. This modular design mitigates hallucinations by transforming passive perception into active, controllable knowledge retrieval
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。