Exception Analysis of Running Complex and Computation-intensive Deep Learning Models

Advances in GPU have facilitated design and execution of complex and computation-intensive deep learning models. As the model complexity increases, the risk of encountering problems due to very large model size, individual tensor size, Not a Number (NaN) value, and memory leak increases as well. When untreated, these problems lead to substantial increase of execution time, generating unpredictable results, and memory leak exceptions. In this paper, we address these problems and particularly large tensor support, C++ kernel changes, and recompilation of the TensorFlow framework. In addition, issues related to NaN value debugging with existing debugging toolkits and solutions to alleviate memory leaks will be explored. Based on experience gained from our analysis, we propose solutions related to better tensor dimension sanity checks, alternative tensor loop procedures, different ways of applying kernels to tensors, a debug trace file filter method, and ways how memory leak exceptions can be resolved. While these problems and solutions may be applicable to running any complex and computation-intensive deep learning model, we described how we encountered them in a use case, in which we designed a deep learning model for activity and gesture recognition using radio data aiming to mitigate domain shift problem.