Reviewing inference performance of state-of-the-art deep learning frameworks

Research output: Contribution to conferencePaperAcademic


Deep learning models have replaced conventional methods for machine learning tasks. Efficient inference on edge devices with limited resources is key for broader deployment. In this work, we focus on the tool selection challenge for inference deployment. We present an extensive evaluation of the inference performance of deep learning software tools using state-of-the-art CNN architectures for multiple hardware platforms. We benchmark these hardware-software pairs for a broad range of network architectures, inference batch sizes, and floating-point precision, focusing on latency and throughput. Our results reveal interesting combinations for optimal tool selection, resulting in different optima when considering minimum latency and maximum throughput.

Original languageEnglish
Number of pages6
Publication statusPublished - 25 May 2020

Fingerprint Dive into the research topics of 'Reviewing inference performance of state-of-the-art deep learning frameworks'. Together they form a unique fingerprint.

Cite this