An open source AutoML benchmark

Pieter Gijsbers; Erin LeDell; Sébastien Poirier; Janek Thomas; Bernd Bischl; Joaquin Vanschoren

An open source AutoML benchmark

Pieter Gijsbers, Erin LeDell, Sébastien Poirier, Janek Thomas, Bernd Bischl, Joaquin Vanschoren

Data Mining

Research output: Contribution to conference › Paper › Academic

558 Downloads (Pure)

Abstract

In recent years, an active field of research has developed around automated machine learning(AutoML). Unfortunately, comparing different AutoML systems is hard and often doneincorrectly. We introduce an open, ongoing, and extensible benchmark framework whichfollows best practices and avoids common mistakes. The framework is open-source, usespublic datasets and has a website with up-to-date results. We use the framework to conducta thorough comparison of 4 AutoML systems across 39 datasets and analyze the results.

Original language	English
Number of pages	8
Publication status	Published - Jun 2019
Event	6th ICML Workshop on Automated Machine Learning - Long Beach, United States Duration: 14 Jun 2019 → 14 Jun 2019 https://sites.google.com/view/automl2019icml/home

Workshop

Workshop	6th ICML Workshop on Automated Machine Learning
Abbreviated title	AutoML@ICML2019
Country/Territory	United States
City	Long Beach
Period	14/06/19 → 14/06/19
Internet address	https://sites.google.com/view/automl2019icml/home

Keywords

AutoML
benchmark
open source

Access to Document

Publishers versionFinal published version, 558 KB

Cite this

@conference{4ae49c3aade448858065f3a0e1bd3cee,

title = "An open source AutoML benchmark",

abstract = "In recent years, an active field of research has developed around automated machine learning(AutoML). Unfortunately, comparing different AutoML systems is hard and often doneincorrectly. We introduce an open, ongoing, and extensible benchmark framework whichfollows best practices and avoids common mistakes. The framework is open-source, usespublic datasets and has a website with up-to-date results. We use the framework to conducta thorough comparison of 4 AutoML systems across 39 datasets and analyze the results.",

keywords = "AutoML, benchmark, open source",

author = "Pieter Gijsbers and Erin LeDell and S{\'e}bastien Poirier and Janek Thomas and Bernd Bischl and Joaquin Vanschoren",

year = "2019",

month = jun,

language = "English",

note = "6th ICML Workshop on Automated Machine Learning, AutoML@ICML2019 ; Conference date: 14-06-2019 Through 14-06-2019",

url = "https://sites.google.com/view/automl2019icml/home",

}

TY - CONF

T1 - An open source AutoML benchmark

AU - Gijsbers, Pieter

AU - LeDell, Erin

AU - Poirier, Sébastien

AU - Thomas, Janek

AU - Bischl, Bernd

AU - Vanschoren, Joaquin

PY - 2019/6

Y1 - 2019/6

N2 - In recent years, an active field of research has developed around automated machine learning(AutoML). Unfortunately, comparing different AutoML systems is hard and often doneincorrectly. We introduce an open, ongoing, and extensible benchmark framework whichfollows best practices and avoids common mistakes. The framework is open-source, usespublic datasets and has a website with up-to-date results. We use the framework to conducta thorough comparison of 4 AutoML systems across 39 datasets and analyze the results.

AB - In recent years, an active field of research has developed around automated machine learning(AutoML). Unfortunately, comparing different AutoML systems is hard and often doneincorrectly. We introduce an open, ongoing, and extensible benchmark framework whichfollows best practices and avoids common mistakes. The framework is open-source, usespublic datasets and has a website with up-to-date results. We use the framework to conducta thorough comparison of 4 AutoML systems across 39 datasets and analyze the results.

KW - AutoML

KW - benchmark

KW - open source

M3 - Paper

T2 - 6th ICML Workshop on Automated Machine Learning

Y2 - 14 June 2019 through 14 June 2019

ER -

An open source AutoML benchmark

Abstract

Workshop

Keywords

Access to Document

Fingerprint

Cite this