用BERT自动分析看图描述中的视觉叙事路径，提升认知障碍评估精度
Advancing Automated Spatio-Semantic Analysis in Picture Description Using Language Models
- 基于BERT的流水线自动提取并排序描述中的内容单元
- 检测准确率达93%精度、96%召回率，序列错误率仅24%
- 结果与人工标注相当，适合临床认知评估场景
当前基于看图描述评估认知语言障碍的方法常忽略视觉叙事路径——说话人描述图片元素的顺序与位置。现有空间语义分析通过内容信息单元（CIUs）捕捉该路径，但依赖人工标注或词典映射，成本高。本研究提出一种基于BERT的流水线，经二元交叉熵与成对排序损失微调，用于从Cookie Theft图片描述中自动提取并排序CIUs。5折交叉验证显示，其在CIU检测上达93%中位数精确率、96%中位数召回率，序列错误率为24%。所提取特征与真实标注具强皮尔逊相关性，外部验证中优于词典基线。这些特征在通过协方差分析比较组间差异时，表现与人工标注特征相当。该方法有效刻画视觉叙事路径，可用于认知障碍评估，代码与模型已开源。
原文摘要 · Abstract (English)
Current methods for automated assessment of cognitive-linguistic impairment via picture description often neglect the visual narrative path - the sequence and locations of elements a speaker described in the picture. Analyses of spatio-semantic features capture this path using content information units (CIUs), but manual tagging or dictionary-based mapping is labor-intensive. This study proposes a BERT-based pipeline, fine tuned with binary cross-entropy and pairwise ranking loss, for automated CIU extraction and ordering from the Cookie Theft picture description. Evaluated by 5-fold cross-validation, it achieves 93% median precision, 96% median recall in CIU detection, and 24% sequence error rates. The proposed method extracts features that exhibit strong Pearson correlations with ground truth, surpassing the dictionary-based baseline in external validation. These features also perform comparably to those derived from manual annotations in evaluating group differences via ANCOVA. The pipeline is shown to effectively characterize visual narrative paths for cognitive impairment assessment, with the implementation and models open-sourced to public.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。