针对视频增强的量化难题,提出分阶段渐进式多帧量化方法。
PMQ-VE: Progressive Multi-Frame Quantization for Video Enhancement
- 采用粗到精两阶段设计,先优化裁剪范围再逐步蒸馏知识。
- 在多个基准上超越现有方法,保持高质量细节且推理更快。
- 适合资源受限设备部署,尤其适用于边缘端视频处理任务。
多帧视频增强旨在利用多帧间的时序信息提升视频的空间与时间分辨率及画质,广泛应用于流媒体处理、监控和生成等领域。尽管基于Transformer的方法性能出色,但其计算与内存开销限制了在边缘设备上的部署。量化通过降低权重和激活的位宽来提升效率,但直接应用现有量化方法常导致性能显著下降,丢失细节。根源在于:(a) 各帧间无法动态分配表示能力,难以自适应动态范围;(b) 过度依赖全精度教师模型,限制低比特学生模型的学习能力。为此,我们提出一种面向视频增强的新型量化方法:渐进式多帧量化(PMQ-VE)。该框架包含两个阶段:基于回溯的多帧量化(BMFQ)与渐进式多教师蒸馏(PMTD)。BMFQ采用百分位初始化与迭代搜索结合剪枝与回溯机制,实现鲁棒的裁剪边界确定;PMTD则通过全精度与多个高比特(INT)教师的渐进蒸馏策略,提升低比特模型的表征能力与质量。大量实验表明,本方法在多个任务与基准上均达到领先性能,优于现有方案。代码将公开于:https://github.com/xiaoBIGfeng/PMQ-VE。
原文摘要 · Abstract (English)
Multi-frame video enhancement tasks aim to improve the spatial and temporal resolution and quality of video sequences by leveraging temporal information from multiple frames, which are widely used in streaming video processing, surveillance, and generation. Although numerous Transformer-based enhancement methods have achieved impressive performance, their computational and memory demands hinder deployment on edge devices. Quantization offers a practical solution by reducing the bit-width of weights and activations to improve efficiency. However, directly applying existing quantization methods to video enhancement tasks often leads to significant performance degradation and loss of fine details. This stems from two limitations: (a) inability to allocate varying representational capacity across frames, which results in suboptimal dynamic range adaptation; (b) over-reliance on full-precision teachers, which limits the learning of low-bit student models. To tackle these challenges, we propose a novel quantization method for video enhancement: Progressive Multi-Frame Quantization for Video Enhancement (PMQ-VE). This framework features a coarse-to-fine two-stage process: Backtracking-based Multi-Frame Quantization (BMFQ) and Progressive Multi-Teacher Distillation (PMTD). BMFQ utilizes a percentile-based initialization and iterative search with pruning and backtracking for robust clipping bounds. PMTD employs a progressive distillation strategy with both full-precision and multiple high-bit (INT) teachers to enhance low-bit models' capacity and quality. Extensive experiments demonstrate that our method outperforms existing approaches, achieving state-of-the-art performance across multiple tasks and benchmarks.The code will be made publicly available at: https://github.com/xiaoBIGfeng/PMQ-VE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。