arXiv:2409.19660cs.CVeess.IV2024-09NeurIPS被引 20

统一编码架构实现人机视觉协同,降低参数开销并保持高性能。

All-in-One Image Coding for Joint Human-Machine Vision with Multi-Path Aggregation

论文配图:All-in-One Image Coding for Joint Human-Machine Vision with Multi-Path Aggregation
图 1 · 摘自论文原文
  • 通过多路径聚合动态分配特征,共享与任务特异性兼顾。
  • 仅需微调1.89%参数即可适配新任务,大幅减少训练成本。
  • 支持人机模式无缝切换,适合跨任务部署的高效编码系统。

面向人眼感知与机器视觉的联合图像编码已受到广泛关注。现有方法通常依赖多个特定任务的编码器-解码器对,导致参数与码率开销高,或在统一表征下难以平衡多目标优化,无法兼顾性能与效率。为此,本文提出在现有编码模型中集成多路径聚合(MPA)机制,构建统一的全功能架构,实现人机视觉联合表征。MPA利用预测器根据任务差异动态分配潜在特征至不同路径,最大化共享特征利用率,同时保留任务特异性特征用于后续优化。基于特征相关性设计两阶段优化策略,缓解多任务性能下降问题。复用共享特征后,仅需对1.89%的参数进行微调即可适配特定任务,完全避免对整个模型的重新优化。实验表明,MPA在人类观看与机器分析任务中,均达到与当前最优方法相当的性能,且在任务特定与多目标优化方面表现均衡。此外,该一体化设计支持人机重建模式的无缝切换,实现无需修改统一模型的任务可控解析。代码开源:https://github.com/NJUVISION/MPA。

原文摘要 · Abstract (English)

Image coding for multi-task applications, catering to both human perception and machine vision, has been extensively investigated. Existing methods often rely on multiple task-specific encoder-decoder pairs, leading to high overhead of parameter and bitrate usage, or face challenges in multi-objective optimization under a unified representation, failing to achieve both performance and efficiency. To this end, we propose Multi-Path Aggregation (MPA) integrated into existing coding models for joint human-machine vision, unifying the feature representation with an all-in-one architecture. MPA employs a predictor to allocate latent features among task-specific paths based on feature importance varied across tasks, maximizing the utility of shared features while preserving task-specific features for subsequent refinement. Leveraging feature correlations, we develop a two-stage optimization strategy to alleviate multi-task performance degradation. Upon the reuse of shared features, as low as 1.89% parameters are further augmented and fine-tuned for a specific task, which completely avoids extensive optimization of the entire model. Experimental results show that MPA achieves performance comparable to state-of-the-art methods in both task-specific and multi-objective optimization across human viewing and machine analysis tasks. Moreover, our all-in-one design supports seamless transitions between human- and machine-oriented reconstruction, enabling task-controllable interpretation without altering the unified model. Code is available at https://github.com/NJUVISION/MPA.

图像编码人机协同多任务学习高效压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。