LOChNES: Modeling Data Movement and On-Chip Network Energy in Neural Network Inference

Abstract

With advancements in computation efficiency, data movement and memory accesses have been regarded as the main bottleneck for reducing AI inference energy consumption. In-Memory Computing has emerged as an approach to address this challenge by amortizing the cost of memory accesses. The memory access and weight writing operations have been well studied and modeled; however, the On-Chip Network (OCN) that facilitates communication between cores has been largely overlooked. The software developed in this work, called \textit{Layer-aware On-Chip Network Evaluation Simulator} (LOChNES), models the OCN communication for a given Convolutional Neural Network on a given hardware realization. Using the aggregated communication distances with the per-distance OCN energy consumption from lab measurements, LOChNES estimates the overall OCN energy consumption for the given CNN. Initial simulation results for ResNet-18 estimate the OCN's contribution to total energy consumption at 15.4%. For ResNet-50, a simple mapping resulted in a 20.2% OCN energy share, while a mapping optimized for cycle count resulted in a 24.6% share. For the latter two, the share of the OCN surpassed the energy consumption of weight loading. It should be noted that, due to limitations in LOChNES's modeling and mapping capabilities, the explored mappings had very low utilizations, at around 9% for ResNet-18 and the simple ResNet-50 mapping, and 13% for the optimized ResNet-50 mapping. Nonetheless, the results show that the optimization of compute energy (this work assumes 100 TOPS/W) has exposed the OCN as the bottleneck for AI inference. While more optimized mapping algorithms could help, circuit and architectural improvements are likely needed.