通过检测样本难易度，让深度网络在早期退出时更可信且更快。
Know What You Don't Know: Selective Prediction for Early Exit DNNs
- 用中间层的判别器判断样本是否难预测，避免仅凭置信度过早退出。
- 相比最终层推理，准确率提升并实现2.05倍加速，误判风险降低50%。
- 适合对可靠性要求高的场景，如医疗、自动驾驶等关键应用。
深度神经网络（DNN）的推理延迟和可信度是其在敏感任务中部署的主要瓶颈。早期退出（EE）DNN通过在中间层允许高置信度样本提前退出来缓解延迟问题。然而，DNN存在过度自信现象，导致大量样本过早退出，使退出策略不可靠。为此，我们采用选择性预测（SP），不依赖单一置信度，而是检查样本的‘难易度’。提出SPEED方法，在每一层引入退避分类器（DC），在决定是否退出前判断样本是否难以预测。若样本难度高，即可能产生幻觉，则将其退避至专家模型。该机制能提前识别困难样本，避免计算资源浪费，并提升推理可信度。实验表明，结合SP的早期退出在保持高精度的同时显著降低延迟。相较于最终层推理，本方法实现2.05倍加速，误判风险降低50%。匿名源码已公开于https://github.com/Div290/SPEED。
原文摘要 · Abstract (English)
Inference latency and trustworthiness of Deep Neural Networks (DNNs) are the bottlenecks in deploying them in critical applications like sensitive tasks. Early Exit (EE) DNNs overcome the latency issues by allowing samples to exit from intermediary layers if they attain `high' confidence scores on the predicted class. However, the DNNs are known to exhibit overconfidence, which can lead to many samples exiting early and render EE strategies untrustworthy. We use Selective Prediction (SP) to overcome this issue by checking the `hardness' of the samples rather than just relying on the confidence score alone. We propose SPEED, a novel approach that uses Deferral Classifiers (DCs) at each layer to check the hardness of samples before performing EEs. Specifically, the DCs identify if a sample is hard to predict at an intermediary layer, leading to hallucination, and defer it to an expert. Early detection of hard samples for inference prevents the wastage of computational resources and improves trust by deferring the hard samples to the expert. We demonstrate that EE aided with SP improves both accuracy and latency. Our method minimizes the risk of wrong prediction by $50\%$ with a speedup of $2.05\times$ as compared to the final layer. The anonymized source code is available at https://github.com/Div290/SPEED
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。