The ABC of data: A classifying framework for data readiness

Laurens A. Castelijns; Yuri Maas; Joaquin Vanschoren

doi:10.1007/978-3-030-43823-4_1

The ABC of data: A classifying framework for data readiness

Laurens A. Castelijns, Yuri Maas, Joaquin Vanschoren

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

9 Citations (Scopus)

208 Downloads (Pure)

Abstract

In order to (semi)automate data cleaning and preprocessing, we need a clear and measurable definition of data quality. Data readiness levels have been proposed to fit this need, but they require a more detailed and measurable definition than is given in prior works. We present a practical framework focused on machine learning that encapsulates data cleaning and (pre)processing procedures. In our framework, datasets are classified within bands, and each band introduces more fine-grained terminology and processing steps. Scores are assigned to each step, resulting in a data quality score. This allows teams of people, as well as automated processes, to track and reason about the cleaning process, and communicate the current status and deficiencies in a more structured, well-documented manner.

Original language	English
Title of host publication	Machine Learning and Knowledge Discovery in Databases - International Workshops of ECML PKDD 2019, Proceedings
Editors	Peggy Cellier, Kurt Driessens
Publisher	Springer
Pages	3-16
Number of pages	14
ISBN (Print)	9783030438227
DOIs	https://doi.org/10.1007/978-3-030-43823-4_1
Publication status	Published - 2020
Event	2019 European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2019) - Wurzburg, Germany Duration: 16 Sept 2019 → 20 Sept 2019 Conference number: 19 http://ecmlpkdd2019.org/

Publication series

Name	Communications in Computer and Information Science
Volume	1167 CCIS
ISSN (Print)	1865-0929
ISSN (Electronic)	1865-0937

Conference

Conference	2019 European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2019)
Abbreviated title	ECML PKDD 2019
Country/Territory	Germany
City	Wurzburg
Period	16/09/19 → 20/09/19
Internet address	http://ecmlpkdd2019.org/

Keywords

Automated data science
Data cleaning
Data quality
Data readiness levels
Preprocessing

Access to Document

10.1007/978-3-030-43823-4_1

The ABC of Data: A Classifying Framework for Data ReadinessFinal author version , 577 KBLicence: CC BY

Cite this

Castelijns, L. A., Maas, Y., & Vanschoren, J. (2020). The ABC of data: A classifying framework for data readiness. In P. Cellier, & K. Driessens (Eds.), Machine Learning and Knowledge Discovery in Databases - International Workshops of ECML PKDD 2019, Proceedings (pp. 3-16). (Communications in Computer and Information Science; Vol. 1167 CCIS). Springer. https://doi.org/10.1007/978-3-030-43823-4_1

@inproceedings{fd3b626d0d4c45ecb919be9944203707,

title = "The ABC of data: A classifying framework for data readiness",

abstract = "In order to (semi)automate data cleaning and preprocessing, we need a clear and measurable definition of data quality. Data readiness levels have been proposed to fit this need, but they require a more detailed and measurable definition than is given in prior works. We present a practical framework focused on machine learning that encapsulates data cleaning and (pre)processing procedures. In our framework, datasets are classified within bands, and each band introduces more fine-grained terminology and processing steps. Scores are assigned to each step, resulting in a data quality score. This allows teams of people, as well as automated processes, to track and reason about the cleaning process, and communicate the current status and deficiencies in a more structured, well-documented manner.",

keywords = "Automated data science, Data cleaning, Data quality, Data readiness levels, Preprocessing",

author = "Castelijns, {Laurens A.} and Yuri Maas and Joaquin Vanschoren",

year = "2020",

doi = "10.1007/978-3-030-43823-4_1",

language = "English",

isbn = "9783030438227",

series = "Communications in Computer and Information Science",

publisher = "Springer",

pages = "3--16",

editor = "Peggy Cellier and Kurt Driessens",

booktitle = "Machine Learning and Knowledge Discovery in Databases - International Workshops of ECML PKDD 2019, Proceedings",

address = "Germany",

note = "2019 European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2019), ECML PKDD 2019 ; Conference date: 16-09-2019 Through 20-09-2019",

url = "http://ecmlpkdd2019.org/",

}

Castelijns, LA, Maas, Y & Vanschoren, J 2020, The ABC of data: A classifying framework for data readiness. in P Cellier & K Driessens (eds), Machine Learning and Knowledge Discovery in Databases - International Workshops of ECML PKDD 2019, Proceedings. Communications in Computer and Information Science, vol. 1167 CCIS, Springer, pp. 3-16, 2019 European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2019), Wurzburg, Germany, 16/09/19. https://doi.org/10.1007/978-3-030-43823-4_1

The ABC of data: A classifying framework for data readiness. / Castelijns, Laurens A.; Maas, Yuri; Vanschoren, Joaquin.
Machine Learning and Knowledge Discovery in Databases - International Workshops of ECML PKDD 2019, Proceedings. ed. / Peggy Cellier; Kurt Driessens. Springer, 2020. p. 3-16 (Communications in Computer and Information Science; Vol. 1167 CCIS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - The ABC of data

T2 - 2019 European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2019)

AU - Castelijns, Laurens A.

AU - Maas, Yuri

AU - Vanschoren, Joaquin

N1 - Conference code: 19

PY - 2020

Y1 - 2020

N2 - In order to (semi)automate data cleaning and preprocessing, we need a clear and measurable definition of data quality. Data readiness levels have been proposed to fit this need, but they require a more detailed and measurable definition than is given in prior works. We present a practical framework focused on machine learning that encapsulates data cleaning and (pre)processing procedures. In our framework, datasets are classified within bands, and each band introduces more fine-grained terminology and processing steps. Scores are assigned to each step, resulting in a data quality score. This allows teams of people, as well as automated processes, to track and reason about the cleaning process, and communicate the current status and deficiencies in a more structured, well-documented manner.

AB - In order to (semi)automate data cleaning and preprocessing, we need a clear and measurable definition of data quality. Data readiness levels have been proposed to fit this need, but they require a more detailed and measurable definition than is given in prior works. We present a practical framework focused on machine learning that encapsulates data cleaning and (pre)processing procedures. In our framework, datasets are classified within bands, and each band introduces more fine-grained terminology and processing steps. Scores are assigned to each step, resulting in a data quality score. This allows teams of people, as well as automated processes, to track and reason about the cleaning process, and communicate the current status and deficiencies in a more structured, well-documented manner.

KW - Automated data science

KW - Data cleaning

KW - Data quality

KW - Data readiness levels

KW - Preprocessing

UR - http://www.scopus.com/inward/record.url?scp=85083740172&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-43823-4_1

DO - 10.1007/978-3-030-43823-4_1

M3 - Conference contribution

AN - SCOPUS:85083740172

SN - 9783030438227

T3 - Communications in Computer and Information Science

SP - 3

EP - 16

BT - Machine Learning and Knowledge Discovery in Databases - International Workshops of ECML PKDD 2019, Proceedings

A2 - Cellier, Peggy

A2 - Driessens, Kurt

PB - Springer

Y2 - 16 September 2019 through 20 September 2019

ER -

The ABC of data: A classifying framework for data readiness

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this