Exploring Early Exiting Strategies for Deep Neural Networks

Abstract

Deep neural networks (DNNs) are renowned for their accuracy across a spectrumof machine learning tasks but often suffer from prolonged inference times due to their depth. To address this, early exiting strategies have been proposed, allowing predictions to be made at intermediate layers, thus reducing inference time. This thesis explores several methodologies to enhance early exit mechanisms within a single multi-exit DNN architecture on an image classification task. First, we demonstrate that replacing the traditional confidence measure entropy with maximum probability achieves comparable accuracy while substantially reducing inference time. Second, we observe that the total uncertainty used by conventional confidence measures does not consistently reflect true model uncertainty, especially for ambiguous images. To address this, we utilize a Dirichlet framework for neural networks, which assumes the network outputs a Dirichlet distribution of class probabilities rather than a point estimate. We introduce a novel training mechanism that includes both original training data and artificially created ambiguous data by blending training images. This approach allows more ambiguous data to exit early compared to the previous approach. Lastly, we examine two alternative perspectives to formulate early exiting criteria. While our experiments focus on a specific neural network architecture and dataset, our methodologies are designed to be independent of architectures and tasks, rendering potential for wider applicability in deep neural networks.