用大模型分层分析视频语义，实现从段落到主题的精准理解
Scribby: A Multi-Level LLM Framework for Semantic Video Analysis
- 分三步提取视频语义：全篇分析、句子解析、相似句聚类
- 通过上下文提示保持语义连贯性，生成可视觉化的语义热图
- 适合需要细粒度视频结构分析的研究者与教育平台开发者
随着教育平台、录播课程和直播娱乐内容的持续增长，对长视频进行高效且结构化分析的需求日益迫切。尽管现有AI系统基于自动生成的字幕提供高层次视频摘要，但这些方法通常仅能提供粗略概述，缺乏对视频结构、主题演进和语义关系的深入分析，难以满足全面视频理解的需求。本文提出一种基于大语言模型（LLM）的多层级视频摘要框架，兼顾宏观理解与微观语义分析。该框架第一阶段在微观层面索引视频：(1) 分析完整字幕文本，(2) 处理单个句子，(3) 利用大模型作为判别器，按语义相似性对句子进行分组。在句子级处理中，通过将全局字幕分析结果与相邻句子信息整合到每个评估提示中，保留上下文连续性。该框架为视频分析工具提供了可视化语义分块与匹配的基础，支持基于相关性的热图呈现。论文还讨论了框架的局限性及未来扩展方向。
原文摘要 · Abstract (English)
As video content continues to expand across educational platforms, recorded lectures, and live-streamed entertainment, the need for efficient and structured analysis of long-form footage has increased \cite{1}. Although many existing AI programs provide high-level video summaries based on AI-generated transcripts \cite{2,3,4,5}, these approaches are often limited to coarse overviews and lack detailed analysis of a video's structure, thematic progression, and semantic relationships, all of which are required for comprehensive video analysis. This paper proposes an LLM-based video summarization framework that balances macro-level comprehension with micro-level semantic analysis \cite{6,12,13}. The first stage of the process indexes the video at a micro level by (1) analyzing the full transcript, (2) analyzing individual transcript sentences, and (3) grouping these sentences by semantic similarity using an LLM as a judge \cite{6,13}. Contextual continuity is retained during sentence-level processing by incorporating both the global transcript analysis and adjacent sentence information into each evaluation prompt. This framework establishes a foundation for video analysis tools that visualize semantic chunking and semantic matching through relevance-based heatmaps. Limitations and future expansions of the framework are also discussed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。