A topological insight into restricted Boltzmann machines

D.C. Mocanu, E. Mocanu, H.P. Nguyen, M. Gibescu, A. Liotta

Onderzoeksoutput: Bijdrage aan tijdschriftTijdschriftartikelAcademicpeer review

25 Citaties (Scopus)

Uittreksel

Restricted Boltzmann Machines (RBMs) and models derived from them have been successfully used as basic building blocks in deep artificial neural networks for automatic features extraction, unsupervised weights initialization, but also as density estimators. Thus, their generative and discriminative capabilities, but also their computational time are instrumental to a wide range of applications. Our main contribution is to look at RBMs from a topological perspective, bringing insights from network science. Firstly, here we show that RBMs and Gaussian RBMs (GRBMs) are bipartite graphs which naturally have a small-world topology. Secondly, we demonstrate both on synthetic and real-world datasets that by constraining RBMs and GRBMs to a scale-free topology (while still considering local neighborhoods and data distribution), we reduce the number of weights that need to be computed by a few orders of magnitude, at virtually no loss in generative performance. Thirdly, we show that, for a fixed number of weights, our proposed sparse models (which by design have a higher number of hidden neurons) achieve better generative capabilities than standard fully connected RBMs and GRBMs (which by design have a smaller number of hidden neurons), at no additional computational costs.
TaalEngels
Pagina's243-270
Aantal pagina's28
TijdschriftMachine Learning
Volume104
Nummer van het tijdschrift2
DOI's
StatusGepubliceerd - sep 2016

Vingerafdruk

Neurons
Topology
Feature extraction
Neural networks
Costs

Citeer dit

@article{c1346a3354ce4004afa0b003e978c887,
title = "A topological insight into restricted Boltzmann machines",
abstract = "Restricted Boltzmann Machines (RBMs) and models derived from them have been successfully used as basic building blocks in deep artificial neural networks for automatic features extraction, unsupervised weights initialization, but also as density estimators. Thus, their generative and discriminative capabilities, but also their computational time are instrumental to a wide range of applications. Our main contribution is to look at RBMs from a topological perspective, bringing insights from network science. Firstly, here we show that RBMs and Gaussian RBMs (GRBMs) are bipartite graphs which naturally have a small-world topology. Secondly, we demonstrate both on synthetic and real-world datasets that by constraining RBMs and GRBMs to a scale-free topology (while still considering local neighborhoods and data distribution), we reduce the number of weights that need to be computed by a few orders of magnitude, at virtually no loss in generative performance. Thirdly, we show that, for a fixed number of weights, our proposed sparse models (which by design have a higher number of hidden neurons) achieve better generative capabilities than standard fully connected RBMs and GRBMs (which by design have a smaller number of hidden neurons), at no additional computational costs.",
author = "D.C. Mocanu and E. Mocanu and H.P. Nguyen and M. Gibescu and A. Liotta",
year = "2016",
month = "9",
doi = "10.1007/s10994-016-5570-z",
language = "English",
volume = "104",
pages = "243--270",
journal = "Machine Learning",
issn = "0885-6125",
publisher = "Springer",
number = "2",

}

A topological insight into restricted Boltzmann machines. / Mocanu, D.C.; Mocanu, E.; Nguyen, H.P.; Gibescu, M.; Liotta, A.

In: Machine Learning, Vol. 104, Nr. 2, 09.2016, blz. 243-270.

Onderzoeksoutput: Bijdrage aan tijdschriftTijdschriftartikelAcademicpeer review

TY - JOUR

T1 - A topological insight into restricted Boltzmann machines

AU - Mocanu,D.C.

AU - Mocanu,E.

AU - Nguyen,H.P.

AU - Gibescu,M.

AU - Liotta,A.

PY - 2016/9

Y1 - 2016/9

N2 - Restricted Boltzmann Machines (RBMs) and models derived from them have been successfully used as basic building blocks in deep artificial neural networks for automatic features extraction, unsupervised weights initialization, but also as density estimators. Thus, their generative and discriminative capabilities, but also their computational time are instrumental to a wide range of applications. Our main contribution is to look at RBMs from a topological perspective, bringing insights from network science. Firstly, here we show that RBMs and Gaussian RBMs (GRBMs) are bipartite graphs which naturally have a small-world topology. Secondly, we demonstrate both on synthetic and real-world datasets that by constraining RBMs and GRBMs to a scale-free topology (while still considering local neighborhoods and data distribution), we reduce the number of weights that need to be computed by a few orders of magnitude, at virtually no loss in generative performance. Thirdly, we show that, for a fixed number of weights, our proposed sparse models (which by design have a higher number of hidden neurons) achieve better generative capabilities than standard fully connected RBMs and GRBMs (which by design have a smaller number of hidden neurons), at no additional computational costs.

AB - Restricted Boltzmann Machines (RBMs) and models derived from them have been successfully used as basic building blocks in deep artificial neural networks for automatic features extraction, unsupervised weights initialization, but also as density estimators. Thus, their generative and discriminative capabilities, but also their computational time are instrumental to a wide range of applications. Our main contribution is to look at RBMs from a topological perspective, bringing insights from network science. Firstly, here we show that RBMs and Gaussian RBMs (GRBMs) are bipartite graphs which naturally have a small-world topology. Secondly, we demonstrate both on synthetic and real-world datasets that by constraining RBMs and GRBMs to a scale-free topology (while still considering local neighborhoods and data distribution), we reduce the number of weights that need to be computed by a few orders of magnitude, at virtually no loss in generative performance. Thirdly, we show that, for a fixed number of weights, our proposed sparse models (which by design have a higher number of hidden neurons) achieve better generative capabilities than standard fully connected RBMs and GRBMs (which by design have a smaller number of hidden neurons), at no additional computational costs.

U2 - 10.1007/s10994-016-5570-z

DO - 10.1007/s10994-016-5570-z

M3 - Article

VL - 104

SP - 243

EP - 270

JO - Machine Learning

T2 - Machine Learning

JF - Machine Learning

SN - 0885-6125

IS - 2

ER -