A meta-learning recommender system for hyperparameter tuning: Predicting when tuning improves SVM classifiers

Rafael G. Mantovani (Corresponding author), André L.D. Rossi, Edesio Alcobaça, Joaquin Vanschoren, André C.P.L.F. de Carvalho

Research output: Contribution to journalArticleAcademicpeer-review

3 Citations (Scopus)
33 Downloads (Pure)

Abstract

For many machine learning algorithms, predictive performance is critically affected by the hyperparameter values used to train them. However, tuning these hyperparameters can come at a high computational cost, especially on larger datasets, while the tuned settings do not always significantly outperform the default values. This paper proposes a recommender system based on meta-learning to identify exactly when it is better to use default values and when to tune hyperparameters for each new dataset. Besides, an in-depth analysis is performed to understand what they take into account for their decisions, providing useful insights. An extensive analysis of different categories of meta-features, meta-learners, and setups across 156 datasets is performed. Results show that it is possible to accurately predict when tuning will significantly improve the performance of the induced models. The proposed system reduces the time spent on optimization processes, without reducing the predictive performance of the induced models (when compared with the ones obtained using tuned hyperparameters). We also explain the decision-making process of the meta-learners in terms of linear separability-based hypotheses. Although this analysis is focused on the tuning of Support Vector Machines, it can also be applied to other algorithms, as shown in experiments performed with decision trees.

Original languageEnglish
Pages (from-to)193-221
Number of pages29
JournalInformation Sciences
Volume501
DOIs
Publication statusPublished - 1 Oct 2019

Fingerprint

Meta-learning
Hyperparameters
Recommender Systems
Recommender systems
Learning Systems
Tuning
Classifiers
Classifier
Decision trees
Learning algorithms
Support vector machines
Learning systems
Separability
Process Optimization
Decision making
Large Data Sets
Decision tree
Computational Cost
Learning Algorithm
Support Vector Machine

Keywords

  • Hyperparameter tuning
  • Meta-learning
  • Recommender system
  • Support vector machines
  • Tuning recommendation

Cite this

Mantovani, Rafael G. ; Rossi, André L.D. ; Alcobaça, Edesio ; Vanschoren, Joaquin ; de Carvalho, André C.P.L.F. / A meta-learning recommender system for hyperparameter tuning : Predicting when tuning improves SVM classifiers. In: Information Sciences. 2019 ; Vol. 501. pp. 193-221.
@article{0fc57033f5f547a9bb45178d6697fc95,
title = "A meta-learning recommender system for hyperparameter tuning: Predicting when tuning improves SVM classifiers",
abstract = "For many machine learning algorithms, predictive performance is critically affected by the hyperparameter values used to train them. However, tuning these hyperparameters can come at a high computational cost, especially on larger datasets, while the tuned settings do not always significantly outperform the default values. This paper proposes a recommender system based on meta-learning to identify exactly when it is better to use default values and when to tune hyperparameters for each new dataset. Besides, an in-depth analysis is performed to understand what they take into account for their decisions, providing useful insights. An extensive analysis of different categories of meta-features, meta-learners, and setups across 156 datasets is performed. Results show that it is possible to accurately predict when tuning will significantly improve the performance of the induced models. The proposed system reduces the time spent on optimization processes, without reducing the predictive performance of the induced models (when compared with the ones obtained using tuned hyperparameters). We also explain the decision-making process of the meta-learners in terms of linear separability-based hypotheses. Although this analysis is focused on the tuning of Support Vector Machines, it can also be applied to other algorithms, as shown in experiments performed with decision trees.",
keywords = "Hyperparameter tuning, Meta-learning, Recommender system, Support vector machines, Tuning recommendation",
author = "Mantovani, {Rafael G.} and Rossi, {Andr{\'e} L.D.} and Edesio Alcoba{\cc}a and Joaquin Vanschoren and {de Carvalho}, {Andr{\'e} C.P.L.F.}",
year = "2019",
month = "10",
day = "1",
doi = "10.1016/j.ins.2019.06.005",
language = "English",
volume = "501",
pages = "193--221",
journal = "Information Sciences",
issn = "0020-0255",
publisher = "Elsevier",

}

A meta-learning recommender system for hyperparameter tuning : Predicting when tuning improves SVM classifiers. / Mantovani, Rafael G. (Corresponding author); Rossi, André L.D.; Alcobaça, Edesio; Vanschoren, Joaquin; de Carvalho, André C.P.L.F.

In: Information Sciences, Vol. 501, 01.10.2019, p. 193-221.

Research output: Contribution to journalArticleAcademicpeer-review

TY - JOUR

T1 - A meta-learning recommender system for hyperparameter tuning

T2 - Predicting when tuning improves SVM classifiers

AU - Mantovani, Rafael G.

AU - Rossi, André L.D.

AU - Alcobaça, Edesio

AU - Vanschoren, Joaquin

AU - de Carvalho, André C.P.L.F.

PY - 2019/10/1

Y1 - 2019/10/1

N2 - For many machine learning algorithms, predictive performance is critically affected by the hyperparameter values used to train them. However, tuning these hyperparameters can come at a high computational cost, especially on larger datasets, while the tuned settings do not always significantly outperform the default values. This paper proposes a recommender system based on meta-learning to identify exactly when it is better to use default values and when to tune hyperparameters for each new dataset. Besides, an in-depth analysis is performed to understand what they take into account for their decisions, providing useful insights. An extensive analysis of different categories of meta-features, meta-learners, and setups across 156 datasets is performed. Results show that it is possible to accurately predict when tuning will significantly improve the performance of the induced models. The proposed system reduces the time spent on optimization processes, without reducing the predictive performance of the induced models (when compared with the ones obtained using tuned hyperparameters). We also explain the decision-making process of the meta-learners in terms of linear separability-based hypotheses. Although this analysis is focused on the tuning of Support Vector Machines, it can also be applied to other algorithms, as shown in experiments performed with decision trees.

AB - For many machine learning algorithms, predictive performance is critically affected by the hyperparameter values used to train them. However, tuning these hyperparameters can come at a high computational cost, especially on larger datasets, while the tuned settings do not always significantly outperform the default values. This paper proposes a recommender system based on meta-learning to identify exactly when it is better to use default values and when to tune hyperparameters for each new dataset. Besides, an in-depth analysis is performed to understand what they take into account for their decisions, providing useful insights. An extensive analysis of different categories of meta-features, meta-learners, and setups across 156 datasets is performed. Results show that it is possible to accurately predict when tuning will significantly improve the performance of the induced models. The proposed system reduces the time spent on optimization processes, without reducing the predictive performance of the induced models (when compared with the ones obtained using tuned hyperparameters). We also explain the decision-making process of the meta-learners in terms of linear separability-based hypotheses. Although this analysis is focused on the tuning of Support Vector Machines, it can also be applied to other algorithms, as shown in experiments performed with decision trees.

KW - Hyperparameter tuning

KW - Meta-learning

KW - Recommender system

KW - Support vector machines

KW - Tuning recommendation

UR - http://www.scopus.com/inward/record.url?scp=85067004352&partnerID=8YFLogxK

U2 - 10.1016/j.ins.2019.06.005

DO - 10.1016/j.ins.2019.06.005

M3 - Article

AN - SCOPUS:85067004352

VL - 501

SP - 193

EP - 221

JO - Information Sciences

JF - Information Sciences

SN - 0020-0255

ER -