通过知识蒸馏提升图像描述的早期退出效率，兼顾速度与准确率。
CAPEEN: Image Captioning with Early Exits and Knowledge Distillation
- 在中间层设置置信度阈值，满足则提前输出结果
- 在MS COCO上实现1.77倍加速，性能接近最终层
- 自适应版本可应对数据分布漂移，适合实际部署
深度神经网络在图像描述任务中取得了显著进展，但其性能提升伴随计算负担和推理延迟增加。早期退出（EE）策略可提高效率，但在图像描述中因需不同层次的语义信息而面临挑战。为此，我们提出CAPEEN，通过知识蒸馏优化EE策略。推理过程中，若预测置信度超过训练阶段学习的预设阈值，则在中间层完成输出。为应对实际部署中目标分布偏离训练数据的问题，我们引入变体A-CAPEEN，利用多臂赌博机框架动态调整阈值。在MS COCO和Flickr30k数据集上的实验表明，CAPEEN在保持与最终层相当性能的同时实现1.77倍加速；A-CAPEEN还具备对分布偏移的鲁棒性。源代码已公开于https://github.com/Div290/CapEEN。
原文摘要 · Abstract (English)
Deep neural networks (DNNs) have made significant progress in recognizing visual elements and generating descriptive text in image-captioning tasks. However, their improved performance comes from increased computational burden and inference latency. Early Exit (EE) strategies can be used to enhance their efficiency, but their adaptation presents challenges in image captioning as it requires varying levels of semantic information for accurate predictions. To overcome this, we introduce CAPEEN to improve the performance of EE strategies using knowledge distillation. Inference in CAPEEN is completed at intermediary layers if prediction confidence exceeds a predefined value learned from the training data. To account for real-world deployments, where target distributions could drift from that of training samples, we introduce a variant A-CAPEEN to adapt the thresholds on the fly using Multiarmed bandits framework. Experiments on the MS COCO and Flickr30k datasets show that CAPEEN gains speedup of 1.77x while maintaining competitive performance compared to the final layer, and A-CAPEEN additionally offers robustness against distortions. The source code is available at https://github.com/Div290/CapEEN
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。