用大模型生成视频重要性权重，提升点播与直播的观看体验。
HiVid: LLM-Guided Video Saliency For Content-Aware VOD And Live Streaming
- 用大模型结合局部感知与全局重排，生成精准视频重要性评分。
- 点播场景下准确率提升11.5%，直播场景提升26%，真实用户测试体验提升14.7%。
- 适合需要动态内容感知的流媒体系统研发者参考。
内容感知流媒体需动态、分块的重要度权重以优化主观体验质量（QoE）。但人工标注成本过高，视觉显著性模型泛化能力差。我们提出HiVid，首个利用大语言模型（LLM）作为可扩展人类代理，在点播（VOD）与直播中生成高保真权重的框架。针对三大挑战：（1）为突破LLM模态限制与令牌数上限，提出感知模块，通过局部上下文窗口自回归构建视频连贯理解；（2）针对点播中局部窗口评分不一致问题，设计排序模块，采用新型LLM引导的归并排序算法进行全局重排；（3）针对直播低延迟、无未来信息需求，提出预测模块，基于多模态时间序列模型，包含内容感知注意力与自适应时域，支持异步LLM推理。大量实验表明，HiVid在点播上权重预测准确率比现有最优提升11.5%，直播提升26%。真实用户研究验证其显著提升14.7%的流媒体体验相关性。
原文摘要 · Abstract (English)
Content-aware streaming requires dynamic, chunk-level importance weights to optimize subjective quality of experience (QoE). However, direct human annotation is prohibitively expensive while vision-saliency models generalize poorly. We introduce HiVid, the first framework to leverage Large Language Models (LLMs) as a scalable human proxy to generate high-fidelity weights for both Video-on-Demand (VOD) and live streaming. We address 3 non-trivial challenges: (1) To extend LLMs' limited modality and circumvent token limits, we propose a perception module to assess frames in a local context window, autoregressively building a coherent understanding of the video. (2) For VOD with rating inconsistency across local windows, we propose a ranking module to perform global re-ranking with a novel LLM-guided merge-sort algorithm. (3) For live streaming which requires low-latency, online inference without future knowledge, we propose a prediction module to predict future weights with a multi-modal time series model, which comprises a content-aware attention and adaptive horizon to accommodate asynchronous LLM inference. Extensive experiments show HiVid improves weight prediction accuracy by up to 11.5\% for VOD and 26\% for live streaming over SOTA baselines. Real-world user study validates HiVid boosts streaming QoE correlation by 14.7\%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。