Academic Publications



Replication and Replication Attempts

the repetition of the methods and protocols employed in a study as to determine whether the initial findings are robust and/or generalizable.

Camerer, C., Dreber, A., Holzmeister, F., et al. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behavior, 2, 637-644.

Alogna, V. K., Attaya, M. K., Aucoin, P., Bahník, Š., Birch, S., Bornstein, B., … & McCoy, S. (2014). Contribution to Alonga et al (2014). Registered replication report: Schooler & Engstler-Schooler (1990). Perspectives on Psychological Science, 9(5), 556-578.

Anderson, S. F., & Maxwell, S. E. (2015). There’s more than one way to conduct a replication study: Beyond statistical significance. Psychology Methodologies, 21(1), 1-12.

Allison, D. B., Shiffrin, R. M., & Stodden V. (2018) Reproducibility of research: Issues and proposed remedies. PNAS, 115(11), 2561-2562.

Patil, P., & Parmigiani, G. (2018) Training replicable predictors in multiple studies. PNAS, 115(11), 2578-2583.

Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J., Fiedler, K., … & Wicherts, J. M. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27(2), 108-119.

Bakker, M., & Wicherts, J. M. (2011). The (mis) reporting of statistical results in psychology journals. Behavior Research Methods, 43(3), 666-678.

Begley, C. G. (2013). Reproducibility: six red flags for suspect work. Nature,497(7450), 433-434.

Begley, C. G., & Ioannidis, J. P. (2015). Reproducibility in Science Improving the Standard for Basic and Preclinical Research. Circulation Research, 116(1), 116-126.

Brandt, M.J., IJzerman, H., Dijksterhius, A., Farach, F.J., Geller, J., Giner-Sorolla, R., … van ‘t Veer, A. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217-224.

Brodeur, A., Lé, M., Sangnier, M., & Zylberberg, Y. (2012). Star wars: The empirics strike back. Paris School of Economics Working Paper, (2012-29).

Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., & Munafò, M. R. (2013). Empirical evidence for low reproducibility indicates low pre-study odds. Nature Reviews Neuroscience, 14(12), 877-877.

Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., & Munafò, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365-376.

Cesario, J. (2014). Priming, replication, and the hardest science. Perspectives on Psychological Science, 9(1), 40-48.

Coffman, L. C., & Niederle, M. (2015). Pre-analysis Plans Have Limited Upside, Especially Where Replications Are Feasible. Journal of Economic Perspectives, 29(3): 81-98.

Collins, F. S., & Tabak, L. A. (2014). Policy: NIH plans to enhance reproducibility. Nature, 505(7485), 612-613.

Dijksterhuis, A. (2014). Welcome Back Theory! Perspectives on Psychological Science, 9(1), 72-75.

Donoho, D. L., & Huo, X. (2004). Beamlab and reproducible research. International Journal of Wavelets, Multiresolution and Information Processing,2(04), 391-414.

Doyen, S., Klein, O., Pichon, C., & Cleeremans, A. (2012). Behavioral priming: It’s all in the mind, but whose mind? PloS One, 7(1), e29081.

Duncan, L. E., & Keller, M. C. (2011). A critical review of the first 10 years of candidate gene-by-environment interaction research in psychiatry. Perspectives, 168(10).

Fanelli, D., & Ioannidis, J. P. (2013). US studies may overestimate effect sizes in softer research. Proceedings of the National Academy of Sciences, 110(37), 15031-15036.

Francis, G. (2012). The psychology of replication and replication in psychology. Perspectives on Psychological Science, 7(6), 585-594.

Francis, G. (2012). The psychology of replication and replication in psychology. Perspectives on Psychological Science, 7(6), 585-594.

Frank, M. C., & Saxe, R. (2012). Teaching replication. Perspectives on Psychological Science, 7(6), 600-604.

Galak, J., LeBoeuf, R. A., Nelson, L. D., & Simmons, J. P. (2012). Correcting the past: Failures to replicate psi. Journal of Personality and Social Psychology,103(6), 933.

Galak, J., & Meyvis, T. (2012). You Could Have Just Asked Reply to Francis (2012). Perspectives on Psychological Science, 7(6), 595-596.

Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016). Comment on “Estimating the reproducibility of psychological science”. Science, 351(6277), 1037.

Grahe, J. E., Reifman, A., Hermann, A. D., Walker, M., Oleson, K. C., Nario-Redmond, M., & Wiebe, R. P. (2012). Harnessing the undiscovered resource of student research projects. Perspectives on Psychological Science, 7(6), 605-607.

Hamermesh, D. S. (2007). Viewpoint: Replication in economics. Canadian Journal of Economics/Revue canadienne d’économique, 40(3), 715-733.

Harris, C. R., Coburn, N., Rohrer, D., & Pashler, H. (2013). Two failures to replicate high-performance-goal priming effects. PloS one, 8(8), e72467.

Heller, R., Bogomolov, M., & Benjamini, Y. (2014). Deciding whether follow-up studies have replicated findings in a preliminary large-scale omics study. Proceedings of the National Academy of Sciences, 111(46), 16262-16267.

Hothorn, T., & Leisch, F. (2011). Case studies in reproducibility. Briefings in Bioinformatics, 12(3), 288-300.

Ioannidis, J. P. (2006). Non-replication and inconsistency in the genome-wide association setting. Human Heredity, 64(4), 203-213.

Ioannidis, J. P. (2013). To Replicate or Not to Replicate: The Case of Pharmacogenetic Studies Have Pharmacogenomics Failed, or Do They Just Need Larger-Scale Evidence and More Replication?. Circulation: Cardiovascular Genetics, 6(4), 413-418.

Ioannidis, J. P. (2012). Scientific inbreeding and same-team replication: type D personality as an example. Journal of Psychosomatic Research, 73(6), 408-410.

Ioannidis, J. P., Nosek, B., & Iorns, E. (2012). Reproducibility concerns. Nature Medicine, 18(12), 1736-1737.

Ioannidis, J. P., Ntzani, E. E., Trikalinos, T. A., & Contopoulos-Ioannidis, D. G. (2001). Replication validity of genetic association studies. Nature Genetics,29(3), 306-309.

Iorns, E., Gunn, W., Erath, J., Rodriguez, A., Zhou, J., Benzinou, M., & Reproducibility Initiative. (2014). Replication Attempt:“Effect of BMAP-28 Antimicrobial Peptides on Leishmania Major Promastigote and Amastigote Growth: Role of Leishmanolysin in Parasite Survival”. PloS One, 9(12), e114614.

Jager, L. R., & Leek, J. T. (2014). An estimate of the science-wise false discovery rate and application to the top medical literature. Biostatistics, 15(1), 1-12.

Jasny, B. R., Chin, G., Chong, L., & Vignieri, S. (2011). Again, and again, and again…. Science, 334(6060), 1225-1225.

Johnson, D. J., Cheung, F., & Donnellan, M. B. (2014). Does cleanliness influence moral judgments? A direct replication of Schnall, Benton, and Harvey (2008). Social Psychology, 45(3), 209.

King, G. (1995). Replication, replication. PS: Political Science & Politics, 28(03), 444-452.

Klein, R.A., Ratliff, K.A., Vianello, M., Adams Jr., R.B., Bahnik, S., Bernstein, M.J., et al. (2014). Investigating variation in replicability: A “many labs” replication project. Social Psychology, 45, 142-152.

Laine, C., Guallar, E., Mulrow, C., Taichman, D. B., Cornell, J. E., Cotton, D., … & Goodman, S. N. (2013). Closing in on the truth about recombinant human bone morphogenetic protein-2: Evidence synthesis, data sharing, peer review, and reproducible research. Annals of Internal Medicine, 158(12), 916-918.

Leggett, N. C., Thomas, N. A., Loetscher, T., & Nicholls, M. E. (2013). The life of p:“Just significant” results are on the rise. The Quarterly Journal of Experimental Psychology, 66(12), 2303-2309.

Li, Q., Brown, J. B., Huang, H., & Bickel, P. J. (2011). Measuring reproducibility of high-throughput experiments. The Annals of Applied Statistics, 5(3), 1752-1779.

Koole, S. L., & Lakens, D. (2012). Rewarding replications a sure and simple way to improve psychological science. Perspectives on Psychological Science, 7(6), 608-614.

Madden, C. S., Easley, R. W., & Dunn, M. G. (1995). How journal editors view replication research. Journal of Advertising, 24(4), 77-87.

Makel, M. C., & Plucker, J. A. (2014). Facts Are More Important Than Novelty Replication in the Education Sciences. Educational Researcher, 0013189X14545513.

Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in Psychology Research How Often Do They Really Occur?. Perspectives on Psychological Science, 7(6), 537-542.

Masicampo, E. J., & Lalande, D. R. (2012). A peculiar prevalence of p values just below. 05. The Quarterly Journal of Experimental Psychology, 65(11), 2271-2279.

Molina, H., Parmigiani, G., & Pandey, A. (2005). Assessing reproducibility of a protein dynamics study using in vivo labeling and liquid chromatography tandem mass spectrometry. Analytical Chemistry, 77(9), 2739-2744.

Neuliep, J. W., & Crandall, R. (1993). Reviewer bias against replication research. Journal of Social Behavior & Personality.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.

Pashler, H., Coburn, N., & Harris, C. R. (2012). Priming of social distance? Failure to replicate effects on social and food judgments. PloS One, 7(8), e42510.

Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7(6), 531-536.

Pashler, H., & Wagenmakers, E. J. (2012). Editors’ Introduction to the Special Section on Replicability in Psychological Science A Crisis of Confidence?. Perspectives on Psychological Science, 7(6), 528-530.

Peers, I. S., Ceuppens, P. R., & Harbron, C. (2012). In search of preclinical robustness. Nature Reviews Drug Discovery, 11(10), 733-734.

Pereira, T. V., & Ioannidis, J. P. (2011). Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects. Journal of Clinical Epidemiology, 64(10), 1060-1069.

Poldrack, R. A., & Poline, J. B. (2014). The publication and reproducibility challenges of shared data. Trends in Cognitive Sciences.

Prinz, F., Schlange, T., & Asadullah, K. (2011). Believe it or not: how much can we rely on published data on potential drug targets? Nature Reviews Drug Discovery, 10(9), 712-712.

Roediger, H. L. (2012). Psychology’s woes and a partial cure: The value of replication. APS Observer, 25(2), 9.

Rohrer, D., Pashler, H., & Harris, C. R. (2015). Do subtle reminders of money change people’s political views? Journal of Experimental Psychology: General, 144(4), e73–e85.

Schmidt, S. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology,13(2), 90.

Simons, D. J. (2014). The value of direct replication. Perspectives on Psychological Science, 9(1), 76-80.

Simons, D. J., Holcombe, A. O., & Spellman, B. A. (2014). An Introduction to Registered Replication Reports at Perspectives on Psychological Science. Perspectives on Psychological Science, 9(5), 552-555.

Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). p-Curve and Effect Size Correcting for Publication Bias Using Only Significant Results. Perspectives on Psychological Science, 9(6), 666-681.

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 0956797611417632.

Siontis, G. C., Tzoulaki, I., Castaldi, P. J., & Ioannidis, J. P. (2015). External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination. Journal of Clinical Epidemiology, 68(1), 25-34.

Stodden, V. (2009). Enabling reproducible research: Open licensing for scientific innovation. International Journal of Communications Law and Policy, Forthcoming.

Stodden, V. (2009). The legal framework for reproducible scientific research: Licensing and copyright. Computing in Science & Engineering, 11(1), 35-40.

Stodden, V., Borwein, J., & Bailey, D. H. (2013). Setting the default to reproducible. Computational Science Research. SIAM News, 46, 4-6.

Stodden, V., Leisch, F., & Peng, R. D. (Eds.). (2014). Implementing Reproducible Research. CRC Press.

Stroebe, W., & Strack, F. (2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9(1), 59-71.

Vandewalle, P., Kovacevic, J., & Vetterli, M. (2009). Reproducible research in signal processing. Signal Processing Magazine, IEEE, 26(3), 37-47.

Waldron, L., Haibe-Kains, B., Culhane, A. C., Riester, M., Ding, J., Wang, X. V., … & Parmigiani, G. (2014). Comparative meta-analysis of prognostic gene signatures for late-stage ovarian cancer. Journal of the National Cancer Institute, 106(5), dju049.

Wicherts, J.M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS One, 6, e26828.

Yong, E. (2012). In the wake of high profile controversies, psychologists are facing up to problems with replication. Nature, 483, 298-300.

BPS invites readers to send (to relevant papers and links to add to this website.