Statistical Forensics



Statistical forensics refer to statistical techniques that can assess the credibility or likely replicability of scientific studies.

Questionable Methods


Science isn’t broken. Fivethirtyeight

Hack your way to scientific glory – FiveThirtyEight Simulation

Bruns, S. B., & Ioannidis, J. P. (2016). p-Curve and p-Hacking in Observational Research. PloS one, 11(2), e0149144.

Costello, V. (2015).“P-hacking”: Megan Head on why it’s bad for science. PLOSblogs./p>

Coy, P. (2017). Investors always think they’re getting ripped off. Here’s why they’re right. BloombergBusinessweek.

Cumming, G. (2016). One reason so many scientific studies may be wrong. The Conversation blog.

Dean, T. (2017). How we edit science part 2: significance testing, p-hacking and peer review. The Conversation blog.

Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The extent and consequences of p-hacking in science. PLoS Biol, 13(3), e1002106.

How much do we know about p-hacking “in the wild”? (2016). Cross Validated.

JeanFrancoisPuget (2016).Green dice are loaded (welcome to p-hacking). IT Best Kept Secret is Optimization blog.

Moody, O. (2017). Psychologist in the soup over food claims. The Times.

Nelson, L.D. (2014). False-positive, p-hacking, statistical power, and evidential value. Berkeley Initiative for Transparency in the Social Sciences.

Neuroskeptic (2015). P-hacking: a talk and further thoughts. Discover Magazine.

Novella, S. (2014). P-hacking and other statistical sins. NEUROLOGICABLOG.

Nuzzo, R. (2014). Statistical errors. i>Nature, 506(7487), 150.

Thieking, M. (2016). John Oliver rips apart bad science on ‘Last Week Tonight’. STAT.

Simpson’s Paradox:

Andale (2013).What is Simpson’s Paradox? Statistics How To.

Carlson, B.W. (2016). Simpson’s paradox. Encyclopedia Britannica, inc.

Hersbein, B. (2015). When average isn’t good enough: Simpson’s Paradox in education and earnings. Brookings.

Pearl, J. (2014). Comment: understanding simpson’s paradox. The American Statistician, 68(1), 8-13.

Simpson’s Paradox

singingbanana (2010). Maths: Simpson’s Paradox. YouTube.

Selective Reporting: Outcomes, Experimental Conditions, and Studies:

Andale (2016). Rporting bias: definition and examples, types. Statistics How To.

Andale (2013). Bias in statistics: definition, selection bias & survivorship bias. Statistics How To.

Bachet, J. & Morduch, J. (2009). Selective knowledge: reporting biases in microfinance data. The Financial Access Initiative.

Butler, N., Delaney, H., & Spoelstra, S. (2017). The Gray Zone: Questionable Research Practices in the Business School. Academy of Management Learning & Education, 16(1), 94-109.

Custer, R. (2013). Research misconduct – the grey area of questionable research practices. VIB.

Dickersin, K., & MIN, Y. I. (1993). Publication bias: the problem that won’t go away. Annals of the New York Academy of Sciences, 703(1), 135-148.

Franco, A., Malhotra, N., & Simonovits, G. (2015). Underreporting in Political Science Survey Experiments: Comparing Questionnaires to Published Results.Political Analysis, 23(2).

Frontiers (2017). Exacerbating the replication crisis in science: replication studies are often unwelcome. PHYS.ORG.

Higgins, J., Altman, D.G., & Sterne, J. (Eds.) (2011). Chapter 8: assessing risk of bias in included studies. In Higgins, J. & Green, S. (Eds.), The Cochrane handbook for systematic reviews of interventions.

John, L.K. (2012). Questionable research practices surprisingly common. Association for Psychological Science.

Mahtani, K. Outcome reporting bias: if you say you’re going to do something, do it! Centre For Evidence-Based Medicine.

Mahtani, K. Outcome reporting bias: is it ok to be a little selective? Centre for Evidence-Based Medicine.

Møller, A. P., & Jennions, M. D. (2001). Testing and adjusting for publication bias. Trends in Ecology & Evolution, 16(10), 580-586.

Norris, S.L., Holmer, H.K., Ogden, L.A., Fu, R.F., Abou-Setta, A.M., Viswanathan, M.S., McPheeters, M.L. (2012). Methods research report: selective outcome reporting as a source of bias in reviews of comparative effectiveness. Agency for Healthcare Research and Quality, Publication No. 12-EHC110-EF.

Outcome Reporting Bias in Trials (ORBIT).

Page, M. J., McKenzie, J. E., Kirkham, J., Dwan, K., Kramer, S., Green, S., & Forbes, A. (2014). Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions. The Cochrane Library.

Questionable research practices: definition, detect, and recommendations for better practices (2015). Replicability-Index.

Rennie, D., & Flanagin, A. (1992). Publication bias: the triumph of hope over experience. Jama, 267(3), 411-412.

Reporting biases. Cochrane Methods Bias.

Salandra, R. (2015). Selective reporting in industrial research: the effect of innovation, uncertainty of science and competition on firm motivation. Druid academy.

Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods, 17(4), 551-566.

Simmons, J.P., Nelson, L.D., Simonsohn, U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science.

Sterne, J., Egger, M., & Moher, D. (Eds.) (2011). Chapter 10: addressing reporting biases. In Higgins, J. & Green, S. (Eds.), The Cochrane handbook for systematic reviews of interventions.

Thornton, A., & Lee, P. (2000). Publication bias in meta-analysis: its causes and consequences. Journal of clinical epidemiology, 53(2), 207-216.

Image Manipulation

Aldhous, P. & Reich, E.S. (2009).Further doubts over stem cell images. NewScientist, 203(2720), 12.

Anderson, C. (1994). Easy-to-alter digital images raise fears of tampering. Science, 263(5145), 317-318.

Couzin-Frankel, J. (2016).Bringing image manipulation to light. Science Magazine.

Davis, P. (2016). Image manipulation: cleaning up the scholarly record. The Scholarly Kitchen.

Gilbert, N. (2009). Science journals crack down on image manipulation. Nature.

Huang, T.S. & Aizawa, K. (1993). Image processing: some challenging problems. Proceedings of the National Academy of Sciences of the United States of America, 90(21), 9766-9769.

McCook, A. (2016). Don’t trust an image in a scientific paper? Manipulation detective’s company wants to help. Retraction Watch.

Nature Publishing Group (2006). Appreciating data: warts, wrinkles and all. Nature Cell Biology, 8(3), 203.

Noorden, R.V. (2015). The image detective who roots out manuscript flaws. Nature.

Pearson, H. (2005). Image manipulation: CSI: cell biology. Nature, 434, 952-953.

Rossner, M. (2007). Hwang case review committee misses the mark. The Journal of Cell Biology, 176(2), 131.

White, C. (2007). Software makes it easier for journals to spot image manipulation. BMJ: British Medical Journal, 334(7594), 607.

Zelip, B. (2013). Digital image manipulation and scientific publishing The Scholarly Commons.

BPS invites readers to send (to relevant papers and links to add to this website.