用自然语言交互修正声呐鱼群追踪结果，提升生态数据处理效率
Teach a Molmo2Fish: Towards interactive fish tracking with natural language guidance
- 通过多模态大模型实现人机对话式追踪纠错
- 在引导与非引导任务中均显著提升追踪准确率
- 适合生态研究者快速修正自动化追踪结果
计算机视觉正被广泛应用于大型生态数据集的识别任务，但复杂任务如多目标追踪仍具挑战。为将视觉模型融入生态研究流程，研究探索如何通过人机协同让不完美的预测结果更可用。本文提出一种新方法：通过与多模态大语言模型进行对话式交互，实现追踪预测的动态修正，以声呐鱼群追踪数据集为初步验证。我们评估了工具 Molmo2Fish 在引导与非引导任务中的表现，包括自我预测和外部追踪结果的修正。结果显示，Molmo2Fish 在鱼群追踪与轨迹修正任务中表现优异，但仍需改进自然语言指导的融合效果。代码与数据已公开于 https://github.com/tidalove/molmo2fish。
原文摘要 · Abstract (English)
Computer vision is increasingly used to automate recognition tasks in large ecological datasets, but more complex tasks such as multi-object tracking continue to pose challenges. As researchers seek to incorporate vision models in ecology workflows, various lines of research have explored how to make imperfect predictions useful through human-in-the-loop processes. We propose a new approach to working with imperfect tracking predictions through an interactive prediction correction workflow taking place as a conversation with a multimodal large language model, which we tailor to a sonar fish tracking dataset as an initial proof of concept. We investigate the performance of the tool, Molmo2Fish, across guided and unguided tasks, correcting its own predicted tracks and external tracks. We find that Molmo2Fish achieves high performance on fish tracking and track correction tasks, but there is still much room to improve on incorporating natural language guidance. The code and data are publicly available at https://github.com/tidalove/molmo2fish.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。