Have it both ways : from A/B testing to A&B testing with exceptional model mining

Wouter Duivesteijn; Tara Farzami; Thijs Putman; Evertjan Peer; Hilde J.P. Weerts; Jasper N. Adegeest; Gerson Foks; Mykola Pechenizkiy

doi:10.1007/978-3-319-71273-4_10

Have it both ways : from A/B testing to A&B testing with exceptional model mining

Wouter Duivesteijn, Tara Farzami, Thijs Putman, Evertjan Peer, Hilde J.P. Weerts, Jasper N. Adegeest, Gerson Foks, Mykola Pechenizkiy

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

3 Citations (Scopus)

5 Downloads (Pure)

Abstract

In traditional A/B testing, we have two variants of the same product, a pool of test subjects, and a measure of success. In a randomized experiment, each test subject is presented with one of the two variants, and the measure of success is aggregated per variant. The variant of the product associated with the most success is retained, while the other variant is discarded. This, however, presumes that the company producing the products only has enough capacity to maintain one of the two product variants. If more capacity is available, then advanced data science techniques can extract more profit for the company from the A/B testing results. Exceptional Model Mining is one such advanced data science technique, which specializes in identifying subgroups that behave differently from the overall population. Using the association model class for EMM, we can find subpopulations that prefer variant A where the general population prefers variant B, and vice versa. This data science technique is applied on data from StudyPortals, a global study choice platform that ran an A/B test on the design of aspects of their website.

Original language	English
Title of host publication	Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2017, Proceedings
Editors	Michelangelo Ceci, Saso Dzeroski, Donato Malerba, Yasemin Altun, Kamalika Das, Jesse Read, Marinka Zitnik, Jerzy Stefanowski, Taneli Mielikäinen
Place of Publication	Cham
Publisher	Springer
Pages	114-126
Number of pages	13
ISBN (Electronic)	978-3-319-71273-4
ISBN (Print)	978-3-319-71272-7
DOIs	https://doi.org/10.1007/978-3-319-71273-4_10
Publication status	Published - 30 Dec 2017
Event	2017 European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2017) - Skopje, Macedonia, The Former Yugoslav Republic of Duration: 18 Sep 2017 → 22 Sep 2017 http://ecmlpkdd2017.ijs.si/index.html

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	10536 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	2017 European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2017)
Abbreviated title	ECML PKDD 2017
Country/Territory	Macedonia, The Former Yugoslav Republic of
City	Skopje
Period	18/09/17 → 22/09/17
Internet address	http://ecmlpkdd2017.ijs.si/index.html

Keywords

A/B testing
Association
E-commerce
Exceptional Model Mining
Online controlled experiments
Website optimization

Access to Document

10.1007/978-3-319-71273-4_10

Cite this

Duivesteijn, W., Farzami, T., Putman, T., Peer, E., Weerts, H. J. P., Adegeest, J. N., Foks, G., & Pechenizkiy, M. (2017). Have it both ways : from A/B testing to A&B testing with exceptional model mining. In M. Ceci, S. Dzeroski, D. Malerba, Y. Altun, K. Das, J. Read, M. Zitnik, J. Stefanowski, & T. Mielikäinen (Eds.), Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2017, Proceedings (pp. 114-126). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10536 LNAI). Springer. https://doi.org/10.1007/978-3-319-71273-4_10

Duivesteijn, Wouter ; Farzami, Tara ; Putman, Thijs ; Peer, Evertjan ; Weerts, Hilde J.P. ; Adegeest, Jasper N. ; Foks, Gerson ; Pechenizkiy, Mykola. / Have it both ways : from A/B testing to A&B testing with exceptional model mining. Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2017, Proceedings. editor / Michelangelo Ceci ; Saso Dzeroski ; Donato Malerba ; Yasemin Altun ; Kamalika Das ; Jesse Read ; Marinka Zitnik ; Jerzy Stefanowski ; Taneli Mielikäinen. Cham : Springer, 2017. pp. 114-126 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{1ec96bb22a5148b4ae53c80544f160a0,

title = "Have it both ways : from A/B testing to A&B testing with exceptional model mining",

abstract = "In traditional A/B testing, we have two variants of the same product, a pool of test subjects, and a measure of success. In a randomized experiment, each test subject is presented with one of the two variants, and the measure of success is aggregated per variant. The variant of the product associated with the most success is retained, while the other variant is discarded. This, however, presumes that the company producing the products only has enough capacity to maintain one of the two product variants. If more capacity is available, then advanced data science techniques can extract more profit for the company from the A/B testing results. Exceptional Model Mining is one such advanced data science technique, which specializes in identifying subgroups that behave differently from the overall population. Using the association model class for EMM, we can find subpopulations that prefer variant A where the general population prefers variant B, and vice versa. This data science technique is applied on data from StudyPortals, a global study choice platform that ran an A/B test on the design of aspects of their website.",

keywords = "A/B testing, Association, E-commerce, Exceptional Model Mining, Online controlled experiments, Website optimization",

author = "Wouter Duivesteijn and Tara Farzami and Thijs Putman and Evertjan Peer and Weerts, {Hilde J.P.} and Adegeest, {Jasper N.} and Gerson Foks and Mykola Pechenizkiy",

year = "2017",

month = dec,

day = "30",

doi = "10.1007/978-3-319-71273-4_10",

language = "English",

isbn = "978-3-319-71272-7",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "114--126",

editor = "Michelangelo Ceci and Saso Dzeroski and Donato Malerba and Yasemin Altun and Kamalika Das and Jesse Read and Marinka Zitnik and Jerzy Stefanowski and Taneli Mielik{\"a}inen",

booktitle = "Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2017, Proceedings",

address = "Germany",

note = "2017 European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2017), ECML PKDD 2017 ; Conference date: 18-09-2017 Through 22-09-2017",

url = "http://ecmlpkdd2017.ijs.si/index.html",

}

Duivesteijn, W, Farzami, T, Putman, T, Peer, E, Weerts, HJP, Adegeest, JN, Foks, G & Pechenizkiy, M 2017, Have it both ways : from A/B testing to A&B testing with exceptional model mining. in M Ceci, S Dzeroski, D Malerba, Y Altun, K Das, J Read, M Zitnik, J Stefanowski & T Mielikäinen (eds), Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2017, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10536 LNAI, Springer, Cham , pp. 114-126, 2017 European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2017), Skopje, Macedonia, The Former Yugoslav Republic of, 18/09/17. https://doi.org/10.1007/978-3-319-71273-4_10

Have it both ways : from A/B testing to A&B testing with exceptional model mining. / Duivesteijn, Wouter; Farzami, Tara; Putman, Thijs; Peer, Evertjan; Weerts, Hilde J.P.; Adegeest, Jasper N.; Foks, Gerson; Pechenizkiy, Mykola.

Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2017, Proceedings. ed. / Michelangelo Ceci; Saso Dzeroski; Donato Malerba; Yasemin Altun; Kamalika Das; Jesse Read; Marinka Zitnik; Jerzy Stefanowski; Taneli Mielikäinen. Cham : Springer, 2017. p. 114-126 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10536 LNAI).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Have it both ways : from A/B testing to A&B testing with exceptional model mining

AU - Duivesteijn, Wouter

AU - Farzami, Tara

AU - Putman, Thijs

AU - Peer, Evertjan

AU - Weerts, Hilde J.P.

AU - Adegeest, Jasper N.

AU - Foks, Gerson

AU - Pechenizkiy, Mykola

PY - 2017/12/30

Y1 - 2017/12/30

N2 - In traditional A/B testing, we have two variants of the same product, a pool of test subjects, and a measure of success. In a randomized experiment, each test subject is presented with one of the two variants, and the measure of success is aggregated per variant. The variant of the product associated with the most success is retained, while the other variant is discarded. This, however, presumes that the company producing the products only has enough capacity to maintain one of the two product variants. If more capacity is available, then advanced data science techniques can extract more profit for the company from the A/B testing results. Exceptional Model Mining is one such advanced data science technique, which specializes in identifying subgroups that behave differently from the overall population. Using the association model class for EMM, we can find subpopulations that prefer variant A where the general population prefers variant B, and vice versa. This data science technique is applied on data from StudyPortals, a global study choice platform that ran an A/B test on the design of aspects of their website.

AB - In traditional A/B testing, we have two variants of the same product, a pool of test subjects, and a measure of success. In a randomized experiment, each test subject is presented with one of the two variants, and the measure of success is aggregated per variant. The variant of the product associated with the most success is retained, while the other variant is discarded. This, however, presumes that the company producing the products only has enough capacity to maintain one of the two product variants. If more capacity is available, then advanced data science techniques can extract more profit for the company from the A/B testing results. Exceptional Model Mining is one such advanced data science technique, which specializes in identifying subgroups that behave differently from the overall population. Using the association model class for EMM, we can find subpopulations that prefer variant A where the general population prefers variant B, and vice versa. This data science technique is applied on data from StudyPortals, a global study choice platform that ran an A/B test on the design of aspects of their website.

KW - A/B testing

KW - Association

KW - E-commerce

KW - Exceptional Model Mining

KW - Online controlled experiments

KW - Website optimization

UR - http://www.scopus.com/inward/record.url?scp=85040256149&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-71273-4_10

DO - 10.1007/978-3-319-71273-4_10

M3 - Conference contribution

AN - SCOPUS:85040256149

SN - 978-3-319-71272-7

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 114

EP - 126

BT - Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2017, Proceedings

A2 - Ceci, Michelangelo

A2 - Dzeroski, Saso

A2 - Malerba, Donato

A2 - Altun, Yasemin

A2 - Das, Kamalika

A2 - Read, Jesse

A2 - Zitnik, Marinka

A2 - Stefanowski, Jerzy

A2 - Mielikäinen, Taneli

PB - Springer

CY - Cham

T2 - 2017 European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2017)

Y2 - 18 September 2017 through 22 September 2017

ER -

Duivesteijn W, Farzami T, Putman T, Peer E, Weerts HJP, Adegeest JN et al. Have it both ways : from A/B testing to A&B testing with exceptional model mining. In Ceci M, Dzeroski S, Malerba D, Altun Y, Das K, Read J, Zitnik M, Stefanowski J, Mielikäinen T, editors, Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2017, Proceedings. Cham : Springer. 2017. p. 114-126. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-71273-4_10