### Abstract

Sparse neural networks are effective approaches to reduce the resource requirements for the deployment of deep neural networks. Recently, the concept of adaptive sparse connectivity, has emerged to allow training sparse neural networks from scratch by optimizing the sparse structure during training. However, comparing different sparse topologies and determining how sparse topologies evolve during training, especially for the situation in which the sparse structure optimization is involved, remain as challenging open questions. This comparison becomes increasingly complex as the number of possible topological comparisons increases exponentially with the size of networks. In this work, we introduce an approach to understand and compare sparse neural network topologies from the perspective of graph theory. We first propose Neural Network Sparse Topology Distance (NNSTD) to measure the distance between different sparse neural networks. Further, we demonstrate that sparse neural networks can outperform over-parameterized models in terms of performance, even without any further structure optimization. To the end, we also show that adaptive sparse connectivity can always unveil a plenitude of sparse sub-networks with very different topologies which outperform the dense model, by quantifying and comparing their topological evolutionary processes. The latter findings complement the \textit{Lottery Ticket Hypothesis} by showing that there is a much more efficient and robust way to find ``winning tickets". Altogether, our results start enabling a better theoretical understanding of sparse neural networks, and demonstrate the utility of using graph theory to analyze them.

Original language | English |
---|---|

Title of host publication | Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD) 2020. |

Number of pages | 17 |

Publication status | Published - 4 Jun 2020 |

### Keywords

- sparse neural networks
- neural network sparse topology distance
- adaptive sparse connectivity
- graph edit distance
- sparse training

## Fingerprint Dive into the research topics of 'Topological Insights into Sparse Neural Networks'. Together they form a unique fingerprint.

## Cite this

Liu, S., van der Lee, T. T., Yaman, A., Atashgahi, Z., Ferrar, D., Sokar, G., Pechenizkiy, M., & Mocanu, D. C. (2020). Topological Insights into Sparse Neural Networks. In

*Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD) 2020.*[2006.14085] https://arxiv.org/abs/2006.14085