Many Labs 2: Investigating Variation in Replicability Across Samples and Settings

Richard A. Klein; Michelangelo Vianello; Fred Hasselman; Byron G. Adams; Reginald B. Adams; Sinan Alper; Mark Aveyard; Jordan R. Axt; Mayowa T. Babalola; Štěpán Bahník; Rishtee Batra; Mihály Berkics; Michael J. Bernstein; Daniel R. Berry; Olga Bialobrzeska; Evans Dami Binan; Konrad Bocian; Mark J. Brandt; Robert Busching; Anna Cabak Rédei; Huajian Cai; Fanny Cambier; Katarzyna Cantarero; Cheryl L. Carmichael; Francisco Ceric; Jesse Chandler; Jen-Ho Chang; Armand Chatard; Eva E. Chen; Winnee Cheong; David C. Cicero; Sharon Coen; Jennifer A. Coleman; Brian Collisson; Morgan A. Conway; Katherine S. Corker; Paul G. Curran; Fiery Cushman; Zubairu K. Dagona; Ilker Dalgar; Anna Dalla Rosa; William E. Davis; Maaike de Bruijn; Leander De Schutter; Thierry Devos; Marieke de Vries; Canay Doğulu; Nerisa Dozo; Kristin Nicole Dukes; Yarrow Dunham; Kevin Durrheim; Charles R. Ebersole; John E. Edlund; Anja Eller; Alexander Scott English; Carolyn Finck; Natalia Frankowska; Miguel-ángel Freyre; Mike Friedman; Elisa Maria Galliani; Joshua C. Gandi; Tanuka Ghoshal; Steffen R. Giessner; Tripat Gill; Timo Gnambs; Ángel Gómez; Roberto González; Jesse Graham; Jon E. Grahe; Ivan Grahek; Eva G.T. Green; Kakul Hai; Matthew Haigh; Elizabeth L. Haines; Michael P. Hall; Marie E. Heffernan; Joshua A. Hicks; Petr Houdek; Jeffrey R. Huntsinger; Ho Phi Huynh; Hans Ijzerman; Yoel Inbar; Åse H. Innes-ker; William Jiménez-leal; Melissa-sue John; Jennifer A. Joy-gaba; Roza G. Kamiloğlu; Heather Barry Kappes; Serdar Karabati; Haruna Karick; Victor N. Keller; Anna Kende; Nicolas Kervyn; Goran Knežević; Carrie Kovacs; Lacy E. Krueger; German Kurapov; Jamie Kurtz; Daniël Lakens; Ljiljana B. Lazarević; Carmel A. Levitan; Neil A. Lewis; Samuel Lins; Nikolette P. Lipsey; Joy E. Losee; Esther Maassen; Angela T. Maitner; Winfrida Malingumu; Robyn K. Mallett; Satia A. Marotta; Janko Međedović; Fernando Mena-pacheco; Taciano L. Milfont; Wendy L. Morris; Sean C. Murphy; Andriy Myachykov; Nick Neave; Koen Neijenhuijs; Anthony J. Nelson; Félix Neto; Austin Lee Nichols; Aaron Ocampo; Susan L. O’donnell; Haruka Oikawa; Masanori Oikawa; Elsie Ong; Gábor Orosz; Malgorzata Osowiecka; Grant Packard; Rolando Pérez-sánchez; Boban Petrović; Ronaldo Pilati; Brad Pinter; Lysandra Podesta; Gabrielle Pogge; Monique M. H. Pollmann; Abraham M. Rutchick; Patricio Saavedra; Alexander K. Saeri; Erika Salomon; Kathleen Schmidt; Felix D. Schönbrodt; Maciej B. Sekerdej; David Sirlopú; Jeanine L. M. Skorinko; Michael A. Smith; Vanessa Smith-castro; Karin C.H.J. Smolders; Agata Sobkow; Walter Sowden; Philipp Spachtholz; Manini Srivastava; Troy G. Steiner; Jeroen Stouten; Chris N.H. Street; Oskar K. Sundfelt; Stephanie Szeto; Ewa Szumowska; Andrew C.W. Tang; Norbert Tanzer; Morgan J. Tear; Jordan Theriault; Manuela Thomae; David Torres; Jakub Traczyk; Joshua M. Tybur; Adrienn Ujhelyi; Robbie C. M. Van Aert; Marcel A. L. M. Van Assen; Marije Van Der Hulst; Paul A. M. Van Lange; Anna Elisabeth Van ’t Veer; Alejandro Vásquez- Echeverría; Leigh Ann Vaughn; Alexandra Vázquez; Luis Diego Vega; Catherine Verniers; Mark Verschoor; Ingrid P. J. Voermans; Marek A. Vranka; Cheryl Welch; Aaron L. Wichman; Lisa A. Williams; Michael Wood; Julie A. Woodzicka; Marta K. Wronska; Liane Young; John M. Zelenski; Zeng Zhijia; Brian A. Nosek

doi:10.1177/2515245918810225

Many Labs 2: Investigating Variation in Replicability Across Samples and Settings

Richard A. Klein, Michelangelo Vianello, Fred Hasselman, Byron G. Adams, Reginald B. Adams, Sinan Alper, Mark Aveyard, Jordan R. Axt, Mayowa T. Babalola, Štěpán Bahník, Rishtee Batra, Mihály Berkics, Michael J. Bernstein, Daniel R. Berry, Olga Bialobrzeska, Evans Dami Binan, Konrad Bocian, Mark J. Brandt, Robert Busching, Anna Cabak RédeiHuajian Cai, Fanny Cambier, Katarzyna Cantarero, Cheryl L. Carmichael, Francisco Ceric, Jesse Chandler, Jen-Ho Chang, Armand Chatard, Eva E. Chen, Winnee Cheong, David C. Cicero, Sharon Coen, Jennifer A. Coleman, Brian Collisson, Morgan A. Conway, Katherine S. Corker, Paul G. Curran, Fiery Cushman, Zubairu K. Dagona, Ilker Dalgar, Anna Dalla Rosa, William E. Davis, Maaike de Bruijn, Leander De Schutter, Thierry Devos, Marieke de Vries, Canay Doğulu, Nerisa Dozo, Kristin Nicole Dukes, Yarrow Dunham, Kevin Durrheim, Charles R. Ebersole, John E. Edlund, Anja Eller, Alexander Scott English, Carolyn Finck, Natalia Frankowska, Miguel-ángel Freyre, Mike Friedman, Elisa Maria Galliani, Joshua C. Gandi, Tanuka Ghoshal, Steffen R. Giessner, Tripat Gill, Timo Gnambs, Ángel Gómez, Roberto González, Jesse Graham, Jon E. Grahe, Ivan Grahek, Eva G.T. Green, Kakul Hai, Matthew Haigh, Elizabeth L. Haines, Michael P. Hall, Marie E. Heffernan, Joshua A. Hicks, Petr Houdek, Jeffrey R. Huntsinger, Ho Phi Huynh, Hans Ijzerman, Yoel Inbar, Åse H. Innes-ker, William Jiménez-leal, Melissa-sue John, Jennifer A. Joy-gaba, Roza G. Kamiloğlu, Heather Barry Kappes, Serdar Karabati, Haruna Karick, Victor N. Keller, Anna Kende, Nicolas Kervyn, Goran Knežević, Carrie Kovacs, Lacy E. Krueger, German Kurapov, Jamie Kurtz, Daniël Lakens, Ljiljana B. Lazarević, Carmel A. Levitan, Neil A. Lewis, Samuel Lins, Nikolette P. Lipsey, Joy E. Losee, Esther Maassen, Angela T. Maitner, Winfrida Malingumu, Robyn K. Mallett, Satia A. Marotta, Janko Međedović, Fernando Mena-pacheco, Taciano L. Milfont, Wendy L. Morris, Sean C. Murphy, Andriy Myachykov, Nick Neave, Koen Neijenhuijs, Anthony J. Nelson, Félix Neto, Austin Lee Nichols, Aaron Ocampo, Susan L. O’donnell, Haruka Oikawa, Masanori Oikawa, Elsie Ong, Gábor Orosz, Malgorzata Osowiecka, Grant Packard, Rolando Pérez-sánchez, Boban Petrović, Ronaldo Pilati, Brad Pinter, Lysandra Podesta, Gabrielle Pogge, Monique M. H. Pollmann, Abraham M. Rutchick, Patricio Saavedra, Alexander K. Saeri, Erika Salomon, Kathleen Schmidt, Felix D. Schönbrodt, Maciej B. Sekerdej, David Sirlopú, Jeanine L. M. Skorinko, Michael A. Smith, Vanessa Smith-castro, Karin C.H.J. Smolders, Agata Sobkow, Walter Sowden, Philipp Spachtholz, Manini Srivastava, Troy G. Steiner, Jeroen Stouten, Chris N.H. Street, Oskar K. Sundfelt, Stephanie Szeto, Ewa Szumowska, Andrew C.W. Tang, Norbert Tanzer, Morgan J. Tear, Jordan Theriault, Manuela Thomae, David Torres, Jakub Traczyk, Joshua M. Tybur, Adrienn Ujhelyi, Robbie C. M. Van Aert, Marcel A. L. M. Van Assen, Marije Van Der Hulst, Paul A. M. Van Lange, Anna Elisabeth Van ’t Veer, Alejandro Vásquez- Echeverría, Leigh Ann Vaughn, Alexandra Vázquez, Luis Diego Vega, Catherine Verniers, Mark Verschoor, Ingrid P. J. Voermans, Marek A. Vranka, Cheryl Welch, Aaron L. Wichman, Lisa A. Williams, Michael Wood, Julie A. Woodzicka, Marta K. Wronska, Liane Young, John M. Zelenski, Zeng Zhijia, Brian A. Nosek

Human Technology Interaction

Research output: Contribution to journal › Article › Academic › peer-review

628 Citations (Scopus)

151 Downloads (Pure)

Abstract

We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance (p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion (p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely high-powered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied.

Original language	English
Pages (from-to)	443-490
Number of pages	48
Journal	Advances in Methods and Practices in Psychological Science
Volume	1
Issue number	4
DOIs	https://doi.org/10.1177/2515245918810225
Publication status	Published - 1 Dec 2018

Keywords

Cognitive psychology
Culture
Individual differences
Meta-analysis
Open data
Open materials
Preregistered
Registered Report
Replication
Sampling effects
Situational effects
Social psychology

Access to Document

10.1177/2515245918810225

2515245918810225Final published version, 1.29 MB

Cite this

Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., Batra, R., Berkics, M., Bernstein, M. J., Berry, D. R., Bialobrzeska, O., Binan, E. D., Bocian, K., Brandt, M. J., Busching, R., ... Nosek, B. A. (2018). Many Labs 2: Investigating Variation in Replicability Across Samples and Settings. Advances in Methods and Practices in Psychological Science, 1(4), 443-490. https://doi.org/10.1177/2515245918810225

@article{dd49c365c06b4bbe88805733bb33a0d2,

title = "Many Labs 2: Investigating Variation in Replicability Across Samples and Settings",

abstract = "We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance (p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion (p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely high-powered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen{\textquoteright}s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied.",

keywords = "Cognitive psychology, Culture, Individual differences, Meta-analysis, Open data, Open materials, Preregistered, Registered Report, Replication, Sampling effects, Situational effects, Social psychology",

author = "Klein, {Richard A.} and Michelangelo Vianello and Fred Hasselman and Adams, {Byron G.} and Adams, {Reginald B.} and Sinan Alper and Mark Aveyard and Axt, {Jordan R.} and Babalola, {Mayowa T.} and {\v S}t{\v e}p{\'a}n Bahn{\'i}k and Rishtee Batra and Mih{\'a}ly Berkics and Bernstein, {Michael J.} and Berry, {Daniel R.} and Olga Bialobrzeska and Binan, {Evans Dami} and Konrad Bocian and Brandt, {Mark J.} and Robert Busching and R{\'e}dei, {Anna Cabak} and Huajian Cai and Fanny Cambier and Katarzyna Cantarero and Carmichael, {Cheryl L.} and Francisco Ceric and Jesse Chandler and Jen-Ho Chang and Armand Chatard and Chen, {Eva E.} and Winnee Cheong and Cicero, {David C.} and Sharon Coen and Coleman, {Jennifer A.} and Brian Collisson and Conway, {Morgan A.} and Corker, {Katherine S.} and Curran, {Paul G.} and Fiery Cushman and Dagona, {Zubairu K.} and Ilker Dalgar and {Dalla Rosa}, Anna and Davis, {William E.} and {de Bruijn}, Maaike and {De Schutter}, Leander and Thierry Devos and {de Vries}, Marieke and Canay Doğulu and Nerisa Dozo and Dukes, {Kristin Nicole} and Yarrow Dunham and Kevin Durrheim and Ebersole, {Charles R.} and Edlund, {John E.} and Anja Eller and English, {Alexander Scott} and Carolyn Finck and Natalia Frankowska and Miguel-{\'a}ngel Freyre and Mike Friedman and Galliani, {Elisa Maria} and Gandi, {Joshua C.} and Tanuka Ghoshal and Giessner, {Steffen R.} and Tripat Gill and Timo Gnambs and {\'A}ngel G{\'o}mez and Roberto Gonz{\'a}lez and Jesse Graham and Grahe, {Jon E.} and Ivan Grahek and Green, {Eva G.T.} and Kakul Hai and Matthew Haigh and Haines, {Elizabeth L.} and Hall, {Michael P.} and Heffernan, {Marie E.} and Hicks, {Joshua A.} and Petr Houdek and Huntsinger, {Jeffrey R.} and Huynh, {Ho Phi} and Hans Ijzerman and Yoel Inbar and Innes-ker, {{\AA}se H.} and William Jim{\'e}nez-leal and Melissa-sue John and Joy-gaba, {Jennifer A.} and Kamiloğlu, {Roza G.} and Kappes, {Heather Barry} and Serdar Karabati and Haruna Karick and Keller, {Victor N.} and Anna Kende and Nicolas Kervyn and Goran Kne{\v z}evi{\'c} and Carrie Kovacs and Krueger, {Lacy E.} and German Kurapov and Jamie Kurtz and Dani{\"e}l Lakens and Lazarevi{\'c}, {Ljiljana B.} and Levitan, {Carmel A.} and Lewis, {Neil A.} and Samuel Lins and Lipsey, {Nikolette P.} and Losee, {Joy E.} and Esther Maassen and Maitner, {Angela T.} and Winfrida Malingumu and Mallett, {Robyn K.} and Marotta, {Satia A.} and Janko Me{\d}edovi{\'c} and Fernando Mena-pacheco and Milfont, {Taciano L.} and Morris, {Wendy L.} and Murphy, {Sean C.} and Andriy Myachykov and Nick Neave and Koen Neijenhuijs and Nelson, {Anthony J.} and F{\'e}lix Neto and {Lee Nichols}, Austin and Aaron Ocampo and O{\textquoteright}donnell, {Susan L.} and Haruka Oikawa and Masanori Oikawa and Elsie Ong and G{\'a}bor Orosz and Malgorzata Osowiecka and Grant Packard and Rolando P{\'e}rez-s{\'a}nchez and Boban Petrovi{\'c} and Ronaldo Pilati and Brad Pinter and Lysandra Podesta and Gabrielle Pogge and Pollmann, {Monique M. H.} and Rutchick, {Abraham M.} and Patricio Saavedra and Saeri, {Alexander K.} and Erika Salomon and Kathleen Schmidt and Sch{\"o}nbrodt, {Felix D.} and Sekerdej, {Maciej B.} and David Sirlop{\'u} and Skorinko, {Jeanine L. M.} and Smith, {Michael A.} and Vanessa Smith-castro and Smolders, {Karin C.H.J.} and Agata Sobkow and Walter Sowden and Philipp Spachtholz and Manini Srivastava and Steiner, {Troy G.} and Jeroen Stouten and Street, {Chris N.H.} and Sundfelt, {Oskar K.} and Stephanie Szeto and Ewa Szumowska and Tang, {Andrew C.W.} and Norbert Tanzer and Tear, {Morgan J.} and Jordan Theriault and Manuela Thomae and David Torres and Jakub Traczyk and Tybur, {Joshua M.} and Adrienn Ujhelyi and {Van Aert}, {Robbie C. M.} and {Van Assen}, {Marcel A. L. M.} and {Van Der Hulst}, Marije and {Van Lange}, {Paul A. M.} and {Van {\textquoteright}t Veer}, {Anna Elisabeth} and {V{\'a}squez- Echeverr{\'i}a}, Alejandro and {Ann Vaughn}, Leigh and Alexandra V{\'a}zquez and Vega, {Luis Diego} and Catherine Verniers and Mark Verschoor and Voermans, {Ingrid P. J.} and Vranka, {Marek A.} and Cheryl Welch and Wichman, {Aaron L.} and Williams, {Lisa A.} and Michael Wood and Woodzicka, {Julie A.} and Wronska, {Marta K.} and Liane Young and Zelenski, {John M.} and Zeng Zhijia and Nosek, {Brian A.}",

year = "2018",

month = dec,

day = "1",

doi = "10.1177/2515245918810225",

language = "English",

volume = "1",

pages = "443--490",

journal = "Advances in Methods and Practices in Psychological Science",

issn = "2515-2459",

publisher = "SAGE Publications Ltd",

number = "4",

}

Klein, RA, Vianello, M, Hasselman, F, Adams, BG, Adams, RB, Alper, S, Aveyard, M, Axt, JR, Babalola, MT, Bahník, Š, Batra, R, Berkics, M, Bernstein, MJ, Berry, DR, Bialobrzeska, O, Binan, ED, Bocian, K, Brandt, MJ, Busching, R, Rédei, AC, Cai, H, Cambier, F, Cantarero, K, Carmichael, CL, Ceric, F, Chandler, J, Chang, J-H, Chatard, A, Chen, EE, Cheong, W, Cicero, DC, Coen, S, Coleman, JA, Collisson, B, Conway, MA, Corker, KS, Curran, PG, Cushman, F, Dagona, ZK, Dalgar, I, Dalla Rosa, A, Davis, WE, de Bruijn, M, De Schutter, L, Devos, T, de Vries, M, Doğulu, C, Dozo, N, Dukes, KN, Dunham, Y, Durrheim, K, Ebersole, CR, Edlund, JE, Eller, A, English, AS, Finck, C, Frankowska, N, Freyre, M, Friedman, M, Galliani, EM, Gandi, JC, Ghoshal, T, Giessner, SR, Gill, T, Gnambs, T, Gómez, Á, González, R, Graham, J, Grahe, JE, Grahek, I, Green, EGT, Hai, K, Haigh, M, Haines, EL, Hall, MP, Heffernan, ME, Hicks, JA, Houdek, P, Huntsinger, JR, Huynh, HP, Ijzerman, H, Inbar, Y, Innes-ker, ÅH, Jiménez-leal, W, John, M, Joy-gaba, JA, Kamiloğlu, RG, Kappes, HB, Karabati, S, Karick, H, Keller, VN, Kende, A, Kervyn, N, Knežević, G, Kovacs, C, Krueger, LE, Kurapov, G, Kurtz, J, Lakens, D, Lazarević, LB, Levitan, CA, Lewis, NA, Lins, S, Lipsey, NP, Losee, JE, Maassen, E, Maitner, AT, Malingumu, W, Mallett, RK, Marotta, SA, Međedović, J, Mena-pacheco, F, Milfont, TL, Morris, WL, Murphy, SC, Myachykov, A, Neave, N, Neijenhuijs, K, Nelson, AJ, Neto, F, Lee Nichols, A, Ocampo, A, O’donnell, SL, Oikawa, H, Oikawa, M, Ong, E, Orosz, G, Osowiecka, M, Packard, G, Pérez-sánchez, R, Petrović, B, Pilati, R, Pinter, B, Podesta, L, Pogge, G, Pollmann, MMH, Rutchick, AM, Saavedra, P, Saeri, AK, Salomon, E, Schmidt, K, Schönbrodt, FD, Sekerdej, MB, Sirlopú, D, Skorinko, JLM, Smith, MA, Smith-castro, V, Smolders, KCHJ, Sobkow, A, Sowden, W, Spachtholz, P, Srivastava, M, Steiner, TG, Stouten, J, Street, CNH, Sundfelt, OK, Szeto, S, Szumowska, E, Tang, ACW, Tanzer, N, Tear, MJ, Theriault, J, Thomae, M, Torres, D, Traczyk, J, Tybur, JM, Ujhelyi, A, Van Aert, RCM, Van Assen, MALM, Van Der Hulst, M, Van Lange, PAM, Van ’t Veer, AE, Vásquez- Echeverría, A, Ann Vaughn, L, Vázquez, A, Vega, LD, Verniers, C, Verschoor, M, Voermans, IPJ, Vranka, MA, Welch, C, Wichman, AL, Williams, LA, Wood, M, Woodzicka, JA, Wronska, MK, Young, L, Zelenski, JM, Zhijia, Z & Nosek, BA 2018, 'Many Labs 2: Investigating Variation in Replicability Across Samples and Settings', Advances in Methods and Practices in Psychological Science, vol. 1, no. 4, pp. 443-490. https://doi.org/10.1177/2515245918810225

TY - JOUR

T1 - Many Labs 2: Investigating Variation in Replicability Across Samples and Settings

AU - Klein, Richard A.

AU - Vianello, Michelangelo

AU - Hasselman, Fred

AU - Adams, Byron G.

AU - Adams, Reginald B.

AU - Alper, Sinan

AU - Aveyard, Mark

AU - Axt, Jordan R.

AU - Babalola, Mayowa T.

AU - Bahník, Štěpán

AU - Batra, Rishtee

AU - Berkics, Mihály

AU - Bernstein, Michael J.

AU - Berry, Daniel R.

AU - Bialobrzeska, Olga

AU - Binan, Evans Dami

AU - Bocian, Konrad

AU - Brandt, Mark J.

AU - Busching, Robert

AU - Rédei, Anna Cabak

AU - Cai, Huajian

AU - Cambier, Fanny

AU - Cantarero, Katarzyna

AU - Carmichael, Cheryl L.

AU - Ceric, Francisco

AU - Chandler, Jesse

AU - Chang, Jen-Ho

AU - Chatard, Armand

AU - Chen, Eva E.

AU - Cheong, Winnee

AU - Cicero, David C.

AU - Coen, Sharon

AU - Coleman, Jennifer A.

AU - Collisson, Brian

AU - Conway, Morgan A.

AU - Corker, Katherine S.

AU - Curran, Paul G.

AU - Cushman, Fiery

AU - Dagona, Zubairu K.

AU - Dalgar, Ilker

AU - Dalla Rosa, Anna

AU - Davis, William E.

AU - de Bruijn, Maaike

AU - De Schutter, Leander

AU - Devos, Thierry

AU - de Vries, Marieke

AU - Doğulu, Canay

AU - Dozo, Nerisa

AU - Dukes, Kristin Nicole

AU - Dunham, Yarrow

AU - Durrheim, Kevin

AU - Ebersole, Charles R.

AU - Edlund, John E.

AU - Eller, Anja

AU - English, Alexander Scott

AU - Finck, Carolyn

AU - Frankowska, Natalia

AU - Freyre, Miguel-ángel

AU - Friedman, Mike

AU - Galliani, Elisa Maria

AU - Gandi, Joshua C.

AU - Ghoshal, Tanuka

AU - Giessner, Steffen R.

AU - Gill, Tripat

AU - Gnambs, Timo

AU - Gómez, Ángel

AU - González, Roberto

AU - Graham, Jesse

AU - Grahe, Jon E.

AU - Grahek, Ivan

AU - Green, Eva G.T.

AU - Hai, Kakul

AU - Haigh, Matthew

AU - Haines, Elizabeth L.

AU - Hall, Michael P.

AU - Heffernan, Marie E.

AU - Hicks, Joshua A.

AU - Houdek, Petr

AU - Huntsinger, Jeffrey R.

AU - Huynh, Ho Phi

AU - Ijzerman, Hans

AU - Inbar, Yoel

AU - Innes-ker, Åse H.

AU - Jiménez-leal, William

AU - John, Melissa-sue

AU - Joy-gaba, Jennifer A.

AU - Kamiloğlu, Roza G.

AU - Kappes, Heather Barry

AU - Karabati, Serdar

AU - Karick, Haruna

AU - Keller, Victor N.

AU - Kende, Anna

AU - Kervyn, Nicolas

AU - Knežević, Goran

AU - Kovacs, Carrie

AU - Krueger, Lacy E.

AU - Kurapov, German

AU - Kurtz, Jamie

AU - Lakens, Daniël

AU - Lazarević, Ljiljana B.

AU - Levitan, Carmel A.

AU - Lewis, Neil A.

AU - Lins, Samuel

AU - Lipsey, Nikolette P.

AU - Losee, Joy E.

AU - Maassen, Esther

AU - Maitner, Angela T.

AU - Malingumu, Winfrida

AU - Mallett, Robyn K.

AU - Marotta, Satia A.

AU - Međedović, Janko

AU - Mena-pacheco, Fernando

AU - Milfont, Taciano L.

AU - Morris, Wendy L.

AU - Murphy, Sean C.

AU - Myachykov, Andriy

AU - Neave, Nick

AU - Neijenhuijs, Koen

AU - Nelson, Anthony J.

AU - Neto, Félix

AU - Lee Nichols, Austin

AU - Ocampo, Aaron

AU - O’donnell, Susan L.

AU - Oikawa, Haruka

AU - Oikawa, Masanori

AU - Ong, Elsie

AU - Orosz, Gábor

AU - Osowiecka, Malgorzata

AU - Packard, Grant

AU - Pérez-sánchez, Rolando

AU - Petrović, Boban

AU - Pilati, Ronaldo

AU - Pinter, Brad

AU - Podesta, Lysandra

AU - Pogge, Gabrielle

AU - Pollmann, Monique M. H.

AU - Rutchick, Abraham M.

AU - Saavedra, Patricio

AU - Saeri, Alexander K.

AU - Salomon, Erika

AU - Schmidt, Kathleen

AU - Schönbrodt, Felix D.

AU - Sekerdej, Maciej B.

AU - Sirlopú, David

AU - Skorinko, Jeanine L. M.

AU - Smith, Michael A.

AU - Smith-castro, Vanessa

AU - Smolders, Karin C.H.J.

AU - Sobkow, Agata

AU - Sowden, Walter

AU - Spachtholz, Philipp

AU - Srivastava, Manini

AU - Steiner, Troy G.

AU - Stouten, Jeroen

AU - Street, Chris N.H.

AU - Sundfelt, Oskar K.

AU - Szeto, Stephanie

AU - Szumowska, Ewa

AU - Tang, Andrew C.W.

AU - Tanzer, Norbert

AU - Tear, Morgan J.

AU - Theriault, Jordan

AU - Thomae, Manuela

AU - Torres, David

AU - Traczyk, Jakub

AU - Tybur, Joshua M.

AU - Ujhelyi, Adrienn

AU - Van Aert, Robbie C. M.

AU - Van Assen, Marcel A. L. M.

AU - Van Der Hulst, Marije

AU - Van Lange, Paul A. M.

AU - Van ’t Veer, Anna Elisabeth

AU - Vásquez- Echeverría, Alejandro

AU - Ann Vaughn, Leigh

AU - Vázquez, Alexandra

AU - Vega, Luis Diego

AU - Verniers, Catherine

AU - Verschoor, Mark

AU - Voermans, Ingrid P. J.

AU - Vranka, Marek A.

AU - Welch, Cheryl

AU - Wichman, Aaron L.

AU - Williams, Lisa A.

AU - Wood, Michael

AU - Woodzicka, Julie A.

AU - Wronska, Marta K.

AU - Young, Liane

AU - Zelenski, John M.

AU - Zhijia, Zeng

AU - Nosek, Brian A.

PY - 2018/12/1

Y1 - 2018/12/1

N2 - We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance (p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion (p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely high-powered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied.

AB - We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance (p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion (p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely high-powered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied.

KW - Cognitive psychology

KW - Culture

KW - Individual differences

KW - Meta-analysis

KW - Open data

KW - Open materials

KW - Preregistered

KW - Registered Report

KW - Replication

KW - Sampling effects

KW - Situational effects

KW - Social psychology

UR - http://www.scopus.com/inward/record.url?scp=85116789185&partnerID=8YFLogxK

U2 - 10.1177/2515245918810225

DO - 10.1177/2515245918810225

M3 - Article

SN - 2515-2459

VL - 1

SP - 443

EP - 490

JO - Advances in Methods and Practices in Psychological Science

JF - Advances in Methods and Practices in Psychological Science

IS - 4

ER -

Many Labs 2: Investigating Variation in Replicability Across Samples and Settings

Abstract

Keywords

Access to Document

Other files and links

Fingerprint