Abstract
For many machine learning algorithms, predictive performance is critically affected by the hyperparameter values used to train them. However, tuning these hyperparameters can come at a high computational cost, especially on larger datasets, while the tuned settings do not always significantly outperform the default values. This paper proposes a recommender system based on meta-learning to identify exactly when it is better to use default values and when to tune hyperparameters for each new dataset. Besides, an in-depth analysis is performed to understand what they take into account for their decisions, providing useful insights. An extensive analysis of different categories of meta-features, meta-learners, and setups across 156 datasets is performed. Results show that it is possible to accurately predict when tuning will significantly improve the performance of the induced models. The proposed system reduces the time spent on optimization processes, without reducing the predictive performance of the induced models (when compared with the ones obtained using tuned hyperparameters). We also explain the decision-making process of the meta-learners in terms of linear separability-based hypotheses. Although this analysis is focused on the tuning of Support Vector Machines, it can also be applied to other algorithms, as shown in experiments performed with decision trees.
| Original language | English |
|---|---|
| Pages (from-to) | 193-221 |
| Number of pages | 29 |
| Journal | Information Sciences |
| Volume | 501 |
| DOIs | |
| Publication status | Published - 1 Oct 2019 |
Fingerprint
Keywords
- Hyperparameter tuning
- Meta-learning
- Recommender system
- Support vector machines
- Tuning recommendation
Cite this
}
A meta-learning recommender system for hyperparameter tuning : Predicting when tuning improves SVM classifiers. / Mantovani, Rafael G. (Corresponding author); Rossi, André L.D.; Alcobaça, Edesio; Vanschoren, Joaquin; de Carvalho, André C.P.L.F.
In: Information Sciences, Vol. 501, 01.10.2019, p. 193-221.Research output: Contribution to journal › Article › Academic › peer-review
TY - JOUR
T1 - A meta-learning recommender system for hyperparameter tuning
T2 - Predicting when tuning improves SVM classifiers
AU - Mantovani, Rafael G.
AU - Rossi, André L.D.
AU - Alcobaça, Edesio
AU - Vanschoren, Joaquin
AU - de Carvalho, André C.P.L.F.
PY - 2019/10/1
Y1 - 2019/10/1
N2 - For many machine learning algorithms, predictive performance is critically affected by the hyperparameter values used to train them. However, tuning these hyperparameters can come at a high computational cost, especially on larger datasets, while the tuned settings do not always significantly outperform the default values. This paper proposes a recommender system based on meta-learning to identify exactly when it is better to use default values and when to tune hyperparameters for each new dataset. Besides, an in-depth analysis is performed to understand what they take into account for their decisions, providing useful insights. An extensive analysis of different categories of meta-features, meta-learners, and setups across 156 datasets is performed. Results show that it is possible to accurately predict when tuning will significantly improve the performance of the induced models. The proposed system reduces the time spent on optimization processes, without reducing the predictive performance of the induced models (when compared with the ones obtained using tuned hyperparameters). We also explain the decision-making process of the meta-learners in terms of linear separability-based hypotheses. Although this analysis is focused on the tuning of Support Vector Machines, it can also be applied to other algorithms, as shown in experiments performed with decision trees.
AB - For many machine learning algorithms, predictive performance is critically affected by the hyperparameter values used to train them. However, tuning these hyperparameters can come at a high computational cost, especially on larger datasets, while the tuned settings do not always significantly outperform the default values. This paper proposes a recommender system based on meta-learning to identify exactly when it is better to use default values and when to tune hyperparameters for each new dataset. Besides, an in-depth analysis is performed to understand what they take into account for their decisions, providing useful insights. An extensive analysis of different categories of meta-features, meta-learners, and setups across 156 datasets is performed. Results show that it is possible to accurately predict when tuning will significantly improve the performance of the induced models. The proposed system reduces the time spent on optimization processes, without reducing the predictive performance of the induced models (when compared with the ones obtained using tuned hyperparameters). We also explain the decision-making process of the meta-learners in terms of linear separability-based hypotheses. Although this analysis is focused on the tuning of Support Vector Machines, it can also be applied to other algorithms, as shown in experiments performed with decision trees.
KW - Hyperparameter tuning
KW - Meta-learning
KW - Recommender system
KW - Support vector machines
KW - Tuning recommendation
UR - http://www.scopus.com/inward/record.url?scp=85067004352&partnerID=8YFLogxK
U2 - 10.1016/j.ins.2019.06.005
DO - 10.1016/j.ins.2019.06.005
M3 - Article
AN - SCOPUS:85067004352
VL - 501
SP - 193
EP - 221
JO - Information Sciences
JF - Information Sciences
SN - 0020-0255
ER -