用大模型扩写语言查询，让动作边界预测更准。
LLM-powered Query Expansion for Enhancing Boundary Prediction in Language-driven Action Localization
- 用LLM生成动作起止的文本描述，补充原始查询
- 通过语义相似度与时间距离建模边界概率，提升训练稳定性
- 无需修改现有模型，可直接套用在各类定位任务中
视频中的语言驱动动作定位不仅需要语言与视频片段的语义对齐，还需准确预测动作边界。然而，语言查询通常只描述动作主体内容，缺乏起止边界的细节信息，导致人工标注主观性强，训练数据存在边界不确定性。本文提出两种策略：一、利用大模型生成动作起止阶段的文本描述，以扩充原始查询，提供更细致的边界提示；二、通过计算帧与扩展查询的语义相似度及帧与标注边界帧的时间距离，建模动作边界的概率得分，从而提供更一致的边界监督信号，增强训练对边界不确定性的鲁棒性。方法具备模型无关性，可零成本集成至任意现有语言驱动动作定位模型中。多个数据集上的实验验证了其有效性。
原文摘要 · Abstract (English)
Language-driven action localization in videos requires not only semantic alignment between language query and video segment, but also prediction of action boundaries. However, the language query primarily describes the main content of an action and usually lacks specific details of action start and end boundaries, which increases the subjectivity of manual boundary annotation and leads to boundary uncertainty in training data. In this paper, on one hand, we propose to expand the original query by generating textual descriptions of the action start and end boundaries through LLMs, which can provide more detailed boundary cues for localization and thus reduce the impact of boundary uncertainty. On the other hand, to enhance the tolerance to boundary uncertainty during training, we propose to model probability scores of action boundaries by calculating the semantic similarities between frames and the expanded query as well as the temporal distances between frames and the annotated boundary frames. They can provide more consistent boundary supervision, thus improving the stability of training. Our method is model-agnostic and can be seamlessly and easily integrated into any existing models of language-driven action localization in an off-the-shelf manner. Experimental results on several datasets demonstrate the effectiveness of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。