Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware

Shiwei Liu (Corresponding author), Decebal Mocanu, Amarsagar Reddy Ramapuram Matavalam, Yulong Pei, Mykola Pechenizkiy

Research output: Contribution to journalArticleAcademicpeer-review

21 Citations (Scopus)
16 Downloads (Pure)


Artificial Neural Networks (ANNs) have emerged as hot topics in the research community. Despite the success of ANNs, it is challenging to train and deploy modern ANNs on commodity hardware due to the ever-increasing model size and the unprecedented growth in the data volumes. Particularly for microarray data, the very-high dimensionality and the small number of samples make it difficult for machine learning techniques to handle. Furthermore, specialized hardware such as Graphics Processing Unit (GPU) is expensive. Sparse neural networks are the leading approaches to address these challenges. However, off-the-shelf sparsity inducing techniques either operate from a pre-trained model or enforce the sparse structure via binary masks. The training efficiency of sparse neural networks cannot be obtained practically. In this paper, we introduce a technique allowing us to train truly sparse neural networks with fixed parameter count throughout training. Our experimental results demonstrate that our method can be applied directly to handle high dimensional data, while achieving higher accuracy than the traditional two phases approaches. Moreover, we have been able to create truly sparse MultiLayer Perceptrons (MLPs) models with over one million neurons and to train them on a typical laptop without GPU ( \url{}), this being way beyond what is possible with any state-of-the-art technique.
Original languageEnglish
Pages (from-to)2589-2604
Number of pages16
JournalNeural Computing and Applications
Issue number7
Publication statusPublished - Apr 2021


  • scalable deep learning
  • sparse neural network
  • microarray gene expression
  • Microarray gene expression
  • Adaptive sparse connectivity
  • Sparse evolutionary training (SET)
  • Truly sparse neural networks


Dive into the research topics of 'Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware'. Together they form a unique fingerprint.
  • Selfish Sparse RNN Training

    Liu, S., Mocanu, D. C., Pei, Y. & Pechenizkiy, M., 18 Jul 2021, Proceedings of the 38th International Conference on Machine Learning (ICML2021) . PMLR, Vol. 139. p. 6893--6904 139

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    Open Access

Cite this