Robust importance-weighted cross-validation under sample selection bias

Wouter M. Kouw; Jesse H. Krijthe; Marco Loog

doi:10.1109/MLSP.2019.8918731

Robust importance-weighted cross-validation under sample selection bias

Wouter M. Kouw, Jesse H. Krijthe, Marco Loog

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

4 Citations (Scopus)

96 Downloads (Pure)

Abstract

Cross-validation under sample selection bias can, in principle, be done by importance-weighting the empirical risk. However, the importance-weighted risk estimator produces suboptimal hyperparameter estimates in problem settings where large weights arise with high probability. We study its sampling variance as a function of the training data distribution and introduce a control variate to increase its robustness to problematically large weights.

Original language	English
Title of host publication	2019 IEEE 29th International Workshop on Machine Learning for Signal Processing, MLSP 2019
Place of Publication	Piscataway
Publisher	Institute of Electrical and Electronics Engineers
Number of pages	6
ISBN (Electronic)	978-1-7281-0824-7
DOIs	https://doi.org/10.1109/MLSP.2019.8918731
Publication status	Published - 5 Dec 2019
Event	29th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2019 - University of Pittsburgh, Pittsburgh, United States Duration: 13 Oct 2019 → 16 Dec 2019 Conference number: 29 https://www.ieeemlsp.cc/

Conference

Conference	29th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2019
Abbreviated title	MLSP 2019
Country/Territory	United States
City	Pittsburgh
Period	13/10/19 → 16/12/19
Internet address	https://www.ieeemlsp.cc/

Keywords

Sample selection bias
cross-validation

Access to Document

10.1109/MLSP.2019.8918731

arXiv versionFinal author version , 330 KB

Cite this

@inproceedings{b5afd26c16514878bb3ee489626681dc,

title = "Robust importance-weighted cross-validation under sample selection bias",

abstract = "Cross-validation under sample selection bias can, in principle, be done by importance-weighting the empirical risk. However, the importance-weighted risk estimator produces suboptimal hyperparameter estimates in problem settings where large weights arise with high probability. We study its sampling variance as a function of the training data distribution and introduce a control variate to increase its robustness to problematically large weights.",

keywords = "Sample selection bias, cross-validation",

author = "Kouw, {Wouter M.} and Krijthe, {Jesse H.} and Marco Loog",

year = "2019",

month = dec,

day = "5",

doi = "10.1109/MLSP.2019.8918731",

language = "English",

booktitle = "2019 IEEE 29th International Workshop on Machine Learning for Signal Processing, MLSP 2019",

publisher = "Institute of Electrical and Electronics Engineers",

address = "United States",

note = "29th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2019, MLSP 2019 ; Conference date: 13-10-2019 Through 16-12-2019",

url = "https://www.ieeemlsp.cc/",

}

Kouw, WM, Krijthe, JH & Loog, M 2019, Robust importance-weighted cross-validation under sample selection bias. in 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing, MLSP 2019., 8918731, Institute of Electrical and Electronics Engineers, Piscataway, 29th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2019, Pittsburgh, Pennsylvania, United States, 13/10/19. https://doi.org/10.1109/MLSP.2019.8918731

Robust importance-weighted cross-validation under sample selection bias. / Kouw, Wouter M.; Krijthe, Jesse H.; Loog, Marco.
2019 IEEE 29th International Workshop on Machine Learning for Signal Processing, MLSP 2019. Piscataway: Institute of Electrical and Electronics Engineers, 2019. 8918731.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Robust importance-weighted cross-validation under sample selection bias

AU - Kouw, Wouter M.

AU - Krijthe, Jesse H.

AU - Loog, Marco

N1 - Conference code: 29

PY - 2019/12/5

Y1 - 2019/12/5

N2 - Cross-validation under sample selection bias can, in principle, be done by importance-weighting the empirical risk. However, the importance-weighted risk estimator produces suboptimal hyperparameter estimates in problem settings where large weights arise with high probability. We study its sampling variance as a function of the training data distribution and introduce a control variate to increase its robustness to problematically large weights.

AB - Cross-validation under sample selection bias can, in principle, be done by importance-weighting the empirical risk. However, the importance-weighted risk estimator produces suboptimal hyperparameter estimates in problem settings where large weights arise with high probability. We study its sampling variance as a function of the training data distribution and introduce a control variate to increase its robustness to problematically large weights.

KW - Sample selection bias

KW - cross-validation

UR - http://www.scopus.com/inward/record.url?scp=85077676600&partnerID=8YFLogxK

U2 - 10.1109/MLSP.2019.8918731

DO - 10.1109/MLSP.2019.8918731

M3 - Conference contribution

BT - 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing, MLSP 2019

PB - Institute of Electrical and Electronics Engineers

CY - Piscataway

T2 - 29th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2019

Y2 - 13 October 2019 through 16 December 2019

ER -

Robust importance-weighted cross-validation under sample selection bias

Abstract

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this