Online function minimization with convex random relu expansions

Laurens Bliek; Michel Verhaegen; Sander Wahls

doi:10.1109/MLSP.2017.8168109

Online function minimization with convex random relu expansions

Laurens Bliek, Michel Verhaegen, Sander Wahls

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

3 Citations (Scopus)

Abstract

We propose CDONE, a convex version of the DONE algorithm. DONE is a derivative-free online optimization algorithm that uses surrogate modeling with noisy measurements to find a minimum of objective functions that are expensive to evaluate. Inspired by their success in deep learning, CDONE makes use of rectified linear units, together with a nonnegativity constraint to enforce convexity of the surrogate model. This leads to a sparse and cheap to evaluate surrogate model of the unknown optimization objective that is still accurate and that can be minimized with convex optimization algorithms. The CDONE algorithm is demonstrated on a toy example and on the problem of hyper-parameter optimization for a deep learning example on handwritten digit classification.

Original language	English
Title of host publication	2017 IEEE International Workshop on Machine Learning for Signal Processing (MLSP)
Editors	Naonori Ueda, Jen-Tzung Chien, Tomoko Matsui, Jan Larsen, Shinji Watanabe
Publisher	Institute of Electrical and Electronics Engineers
Pages	1-6
Number of pages	6
ISBN (Electronic)	978-1-5090-6341-3
DOIs	https://doi.org/10.1109/MLSP.2017.8168109
Publication status	Published - 7 Dec 2017
Externally published	Yes
Event	27th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2017 - Tokyo, Japan Duration: 25 Sept 2017 → 28 Sept 2017 Conference number: 27

Conference

Conference	27th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2017
Abbreviated title	MLSP 2017
Country/Territory	Japan
City	Tokyo
Period	25/09/17 → 28/09/17

Keywords

Bayesian optimization
Deep learning
Derivative-free optimization
Surrogate modeling

Access to Document

10.1109/MLSP.2017.8168109

Cite this

Bliek, L., Verhaegen, M., & Wahls, S. (2017). Online function minimization with convex random relu expansions. In N. Ueda, J.-T. Chien, T. Matsui, J. Larsen, & S. Watanabe (Eds.), 2017 IEEE International Workshop on Machine Learning for Signal Processing (MLSP) (pp. 1-6). Article 8168109 Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/MLSP.2017.8168109

@inproceedings{b2dbfb04fb7e4482b12e5891086b51e2,

title = "Online function minimization with convex random relu expansions",

abstract = "We propose CDONE, a convex version of the DONE algorithm. DONE is a derivative-free online optimization algorithm that uses surrogate modeling with noisy measurements to find a minimum of objective functions that are expensive to evaluate. Inspired by their success in deep learning, CDONE makes use of rectified linear units, together with a nonnegativity constraint to enforce convexity of the surrogate model. This leads to a sparse and cheap to evaluate surrogate model of the unknown optimization objective that is still accurate and that can be minimized with convex optimization algorithms. The CDONE algorithm is demonstrated on a toy example and on the problem of hyper-parameter optimization for a deep learning example on handwritten digit classification.",

keywords = "Bayesian optimization, Deep learning, Derivative-free optimization, Surrogate modeling",

author = "Laurens Bliek and Michel Verhaegen and Sander Wahls",

year = "2017",

month = dec,

day = "7",

doi = "10.1109/MLSP.2017.8168109",

language = "English",

pages = "1--6",

editor = "Naonori Ueda and Jen-Tzung Chien and Tomoko Matsui and Jan Larsen and Shinji Watanabe",

booktitle = "2017 IEEE International Workshop on Machine Learning for Signal Processing (MLSP)",

publisher = "Institute of Electrical and Electronics Engineers",

address = "United States",

note = "27th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2017, MLSP 2017 ; Conference date: 25-09-2017 Through 28-09-2017",

}

Bliek, L, Verhaegen, M & Wahls, S 2017, Online function minimization with convex random relu expansions. in N Ueda, J-T Chien, T Matsui, J Larsen & S Watanabe (eds), 2017 IEEE International Workshop on Machine Learning for Signal Processing (MLSP)., 8168109, Institute of Electrical and Electronics Engineers, pp. 1-6, 27th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2017, Tokyo, Japan, 25/09/17. https://doi.org/10.1109/MLSP.2017.8168109

Online function minimization with convex random relu expansions. / Bliek, Laurens; Verhaegen, Michel; Wahls, Sander.
2017 IEEE International Workshop on Machine Learning for Signal Processing (MLSP). ed. / Naonori Ueda; Jen-Tzung Chien; Tomoko Matsui; Jan Larsen; Shinji Watanabe. Institute of Electrical and Electronics Engineers, 2017. p. 1-6 8168109.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Online function minimization with convex random relu expansions

AU - Bliek, Laurens

AU - Verhaegen, Michel

AU - Wahls, Sander

N1 - Conference code: 27

PY - 2017/12/7

Y1 - 2017/12/7

N2 - We propose CDONE, a convex version of the DONE algorithm. DONE is a derivative-free online optimization algorithm that uses surrogate modeling with noisy measurements to find a minimum of objective functions that are expensive to evaluate. Inspired by their success in deep learning, CDONE makes use of rectified linear units, together with a nonnegativity constraint to enforce convexity of the surrogate model. This leads to a sparse and cheap to evaluate surrogate model of the unknown optimization objective that is still accurate and that can be minimized with convex optimization algorithms. The CDONE algorithm is demonstrated on a toy example and on the problem of hyper-parameter optimization for a deep learning example on handwritten digit classification.

AB - We propose CDONE, a convex version of the DONE algorithm. DONE is a derivative-free online optimization algorithm that uses surrogate modeling with noisy measurements to find a minimum of objective functions that are expensive to evaluate. Inspired by their success in deep learning, CDONE makes use of rectified linear units, together with a nonnegativity constraint to enforce convexity of the surrogate model. This leads to a sparse and cheap to evaluate surrogate model of the unknown optimization objective that is still accurate and that can be minimized with convex optimization algorithms. The CDONE algorithm is demonstrated on a toy example and on the problem of hyper-parameter optimization for a deep learning example on handwritten digit classification.

KW - Bayesian optimization

KW - Deep learning

KW - Derivative-free optimization

KW - Surrogate modeling

UR - http://www.scopus.com/inward/record.url?scp=85042254911&partnerID=8YFLogxK

U2 - 10.1109/MLSP.2017.8168109

DO - 10.1109/MLSP.2017.8168109

M3 - Conference contribution

AN - SCOPUS:85042254911

SP - 1

EP - 6

BT - 2017 IEEE International Workshop on Machine Learning for Signal Processing (MLSP)

A2 - Ueda, Naonori

A2 - Chien, Jen-Tzung

A2 - Matsui, Tomoko

A2 - Larsen, Jan

A2 - Watanabe, Shinji

PB - Institute of Electrical and Electronics Engineers

T2 - 27th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2017

Y2 - 25 September 2017 through 28 September 2017

ER -

Online function minimization with convex random relu expansions

Abstract

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this