Hierarchical clustering for discrimination discovery: a top-down approach

Neda Nasiriani, Anna Squicciarini, Zara Saldanha, Sanchit Goel, Nicola Zannone

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

Abstract

Today, data is an essential part of many decision-making processes in businesses and social life through the use of various machine learning techniques. These methods can easily perpetuate human bias in the data and result in discrimination. Despite a growing interest in data discrimination discovery and removal, to date there is a lack of a general and robust framework to distinguish discriminatory decision-making processes from non-discriminatory ones. In this work, we present a generic framework that helps detect possible discrimination by analyzing historical data and associated decisions using a top-down unsupervised approach, which we refer to as hierarchical clustering. Our approach is highly adaptive as it gradually 'learns' users' inherent groups, and clusters their records using cohesiveness and density of points in the dataset. Moreover, we propose a progressive attribute-selection method to choose statistically relevant attributes, thus reducing the effect of noise. Finally, we adopt a recursive notion of cluster profile that is homogeneous w.r.t. decision labels. This allows for deeper insights on the data and on the decision-making underlying the final user classification. Our framework is able to identify both positive and negative bias resulting in discrimination. We also highlight patterns of discrimination revealed by the homogeneous cluster centroids, which otherwise could not be captured.

Original languageEnglish
Title of host publicationProceedings - IEEE 2nd International Conference on Artificial Intelligence and Knowledge Engineering, AIKE 2019
Place of PublicationPiscataway
PublisherInstitute of Electrical and Electronics Engineers
Pages187-194
Number of pages8
ISBN (Electronic)978-1-7281-1488-0
DOIs
Publication statusPublished - 1 Jun 2019
Event2nd IEEE International Conference on Artificial Intelligence and Knowledge Engineering, AIKE 2019 - Cagliari, Sardinia, Italy
Duration: 3 Jun 20195 Jun 2019

Conference

Conference2nd IEEE International Conference on Artificial Intelligence and Knowledge Engineering, AIKE 2019
CountryItaly
CityCagliari, Sardinia
Period3/06/195/06/19

Keywords

  • Bias
  • Clustering
  • Discrimination Discovery
  • KNN
  • Measurements

Fingerprint Dive into the research topics of 'Hierarchical clustering for discrimination discovery: a top-down approach'. Together they form a unique fingerprint.

Cite this