Evolving and understanding sparse deep neural networks using cosine similarity

J. Pieterse, Decebal Mocanu

Onderzoeksoutput: Bijdrage aan tijdschriftTijdschriftartikelAcademic

9 Downloads (Pure)

Samenvatting

Training sparse neural networks with adaptive connectivity is an active research topic. Such networks require less storage and have lower computational complexity compared to their dense counterparts. The Sparse Evolutionary Training (SET) procedure uses weights magnitude to evolve efficiently the topology of a sparse network to fit the dataset, while enabling it to have quadratically less parameters than its dense counterpart. To this end, we propose a novel approach that evolves a sparse network topology based on the behavior of neurons in the network. More exactly, the cosine similarities between the activations of any two neurons are used to determine which connections are added to or removed from the network. By integrating our approach within the SET procedure, we propose 5 new algorithms to train sparse neural networks. We argue that our approach has low additional computational complexity and we draw a parallel to Hebbian learning. Experiments are performed on 8 datasets taken from various domains to demonstrate the general applicability of our approach. Even without optimizing hyperparameters for specific datasets, the experiments show that our proposed training algorithms usually outperform SET and state-of-the-art dense neural network techniques. The last but not the least, we show that the evolved connectivity patterns of the input neurons reflect their impact on the classification task.
Originele taal-2Engels
Artikelnummer1903.07138v1
Aantal pagina's14
TijdschriftarXiv
StatusGepubliceerd - 17 mrt 2019

    Vingerafdruk

Citeer dit