An open source AutoML benchmark

Pieter Gijsbers, Erin LeDell, Sébastien Poirier, Janek Thomas, Bernd Bischl, Joaquin Vanschoren

Research output: Contribution to conferencePaperAcademic

529 Downloads (Pure)


In recent years, an active field of research has developed around automated machine learning(AutoML). Unfortunately, comparing different AutoML systems is hard and often doneincorrectly. We introduce an open, ongoing, and extensible benchmark framework whichfollows best practices and avoids common mistakes. The framework is open-source, usespublic datasets and has a website with up-to-date results. We use the framework to conducta thorough comparison of 4 AutoML systems across 39 datasets and analyze the results.
Original languageEnglish
Number of pages8
Publication statusPublished - Jun 2019
Event6th ICML Workshop on Automated Machine Learning - Long Beach, United States
Duration: 14 Jun 201914 Jun 2019


Workshop6th ICML Workshop on Automated Machine Learning
Abbreviated titleAutoML@ICML2019
Country/TerritoryUnited States
CityLong Beach
Internet address


  • AutoML
  • benchmark
  • open source


Dive into the research topics of 'An open source AutoML benchmark'. Together they form a unique fingerprint.

Cite this