用混合模型加速视频编码,兼顾速度与画质。
Accelerating HEVC Intra Partitioning via a CNN-Hierarchical Attention Transformer Hybrid

- 融合卷积与分层注意力机制,高效捕捉局部与全局特征。
- 在不同画质等级下,相比基线降低2.4至7.9个百分点的码率损失。
- 保持低延迟,适合实时视频编码部署。
HEVC的递归四叉树分割带来显著计算开销,其中对编码单元(CTU)分区预测的穷举率失真优化占编码时间主导。尽管深度学习可加速分区预测,但存在架构困境:卷积网络计算高效但视野受限,难以捕捉长程语义关系和重复纹理;而基于变换器的结构虽擅长全局建模,却导致极高CPU延迟,制约其在主流以CPU为主的部署场景应用。本文提出混合快速视觉变换器(HFViT),融合重参数化深度可分离卷积主干与分层注意力变换器(HAT)机制,通过载体令牌方案实现亚二次复杂度下的高效全局信息传播。训练后结构融合将批归一化合并至前序层,进一步降低延迟。全面评估表明,HFViT在多分辨率下均能有效加速HEVC帧内编码。在标准JCT-VC测试序列上,相较于竞争基线ETH-CNN,HFViT在类别A、B、E上的平均VMAF BD-rate惩罚分别降低2.4、2.6、7.9个百分点,同时保持CPU推理延迟低于基准卷积模型的8%,并在GPU上提速40%,验证了其实时编码集成的实际可行性。
原文摘要 · Abstract (English)
The recursive quad-tree partitioning in High Efficiency Video Coding (HEVC) incurs considerable computational overhead, with exhaustive rate-distortion optimization for CTU partition prediction consuming the dominant share of encoding time. Although partition prediction through deep learning has emerged as a viable encoding accelerator, an architectural dichotomy remains largely unaddressed: CNNs are computationally efficient but spatially myopic due to their localized effective receptive fields, failing to capture long range semantic relationships and repetitive textures; conversely, transformer based architectures are better at capturing global context but incur prohibitive CPU latency, a critical liability that impedes deployment which is predominantly CPU-bound. This paper introduces Hybrid Fast Vision Transformer (HFViT), a hybrid architecture designed to accelerate HEVC intra-mode partition prediction. HFViT fuses a reparameterized depthwise-separable convolutional backbone with a Hierarchical Attention Transformer (HAT) mechanism, leveraging a carrier token scheme to enable efficient global information propagation at sub-quadratic complexity. Post-training structural fusion collapses batch normalization into preceding layers to further reduce latency. Comprehensive evaluation reveals the efficacy of HFViT in accelerating HEVC intra-encoding across resolutions. On standard JCT-VC test sequences, HFViT reduces the average VMAF BD-rate penalty by 2.4, 2.6, and 7.9 percentage points on Classes A, B and E, respectively, as compared to the competing ETH-CNN baseline while maintaining CPU inference latency within 8% of the CNN baseline and surpassing it on GPU by 40%, establishing practical viability for real-time encoder integration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。