用目标检测辅助LLM提升计数能力，解决罕见类别与反事实问题。
Solution for OOD-CV UNICORN Challenge 2024 Object Detection Assistance LLM Counting Ability Improvement
- 用目标检测模型为LLM提供物体位置和数量信息，增强推理基础。
- 在挑战赛中取得0.86得分，排名第二，显著提升对罕见类别的泛化能力。
- 适合需要高鲁棒性视觉问答与计数的AI系统开发者参考。
本报告详细描述了我们在ECCV OOD-CV UNICORN Challenge 2024中探索并提出的方案，聚焦于大语言模型在视觉问答任务中的鲁棒性。竞赛数据集包括OODCA-VQA和SketchyQA。为测试模型鲁棒性，组织方扩展了两个变体：OODCV-Counterfactual和Sketchy-Challenging。前者包含具有转折点和计算步骤的复杂问题，后者使用较少见的物品类别以检验模型泛化能力。针对这些挑战，我们提出一种简单而有效的方案——对象检测辅助大语言模型计数能力提升（ODAC），包含两大模块：(1)对象检测辅助；(2)反事实特定提示。该方法在最终测试中取得0.86分，排名第二。
原文摘要 · Abstract (English)
This report provide a detailed description of the method that we explored and proposed in the ECCV OOD-CV UNICORN Challenge 2024, which focusing on the robustness of responses from large language models. The dataset of this competition are OODCA-VQA and SketchyQA. In order to test the robustness of the model. The organizer extended two variants of the dataset OODCV-Counterfactual and Sketchy-Challenging. There are several difficulties with these datasets. Firstly, the Sketchy-Challenging dataset uses some rarer item categories to test the model's generalization ability. Secondly, in the OODCV-Counterfactual dataset, the given problems often have inflection points and computational steps, requiring the model to recognize them during the inference process. In order to address this issue, we propose a simple yet effective approach called Object Detection Assistance Large Language Model(LLM) Counting Ability Improvement(ODAC), which focuses on using the object detection model to assist the LLM. To clarify, our approach contains two main blocks: (1)Object Detection Assistance. (2) Counterfactual Specific prompt. Our approach ranked second in the final test with a score of 0.86.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。