Statistical Forensics

Topics


Statistical forensics refer to statistical techniques that can assess the credibility or likely replicability of scientific studies.


Techniques to Detect and Eliminate Fraud


Checking the Distribution of Rightmost Digits:

Check Rightmost Digits for Uniform Distribution. The Office of Research Integrity

Beber, B., Scacco, A., & Alvarez, R. M. (2012). What the numbers say: A digit-based test for election fraud. Political Analysis, 211-234.

Diekmann, A. (2007). Not the First Digit! Using Benford’s Law to Detect Fraudulent Scientific Data. Journal of Applied Statistics, 34(3), 321-329.

Dlugosz, S., & Müller-Funk, U. (2009). The value of the last digit: statistical fraud detection with digit analysis. Advances in data analysis and classification, 3(3), 281-290.

Mosimann, J., Dahlberg, J., Davidian, N., & Krueger, J. (2002). Terminal digits and the examination of questioned data. Accountability in Research: Policies and Quality Assurance, 9(2), 75-92.445.

Pitt, J. H., & Hill, H. Z. Statistical Detection of Potentially Fabricated Numerical Data: A Case Study.


Testing for Linguistic Obfuscation:

Carey, Bjorn (2015). Stanford researchers uncover patterns in how scientists lie about their data Stanford News

Dalal, F. (2015). Statistical spin: Linguistic obfuscation—The art of overselling the CBT evidence base. The Journal of Psychological Therapies in Primary Care, 4(1), 1-25.

Little, L. (1998). Hiding with Words: Obfuscation, Avoidance, and Federal Jurisdiction Opinions. UCLA Law Review, 46(1), 75-160.

Mairs, M.A., Linguistic Obfuscation Techniques. (Masters Thesis, The University of Liverpool).

Markowitz, D.M. &amp Hancock, J.T. (2015)Linguistic Obfuscation in Fraudulent Science. Journal of Language and Social Psychology, 35(4), 435-445.

Markowitz, D. M., & Hancock, J. T. (2014). Linguistic traces of a scientific fraud: The case of Diederik Stapel. PloS one, 9(8), e105937.

Newman, M. L., Pennebaker, J. W., Berry, D. S., & Richards, J. M. (2003). Lying words: Predicting deception from linguistic styles. Personality and social psychology bulletin, 29(5), 665-675.

Simmons, J. P., & Simonsohn, U. (2017). Power posing: P-curving the evidence. Psychological Science.

Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2015). Better P-curves: Making P-curve analysis more robust to errors, fraud, and ambitious P-hacking, a Reply to Ulrich and Miller (2015).

Tanner, S. (2015). Evidence of false positives in research clearinghouses and influential journals: An application of P-curve to policy research. Observational Studies, 1, 18-29.

Toma, C. L., & Hancock, J. T. (2012). What lies beneath: The linguistic traces of deception in online dating profiles. Journal of Communication, 62(1), 78-97.


P-Curves:

Bishop, D. V., & Thompson, P. A. (2016). Problems in using p-curve analysis and text-mining to detect rate of p-hacking and evidential value. PeerJ, 4(1715).

Enserink, M. (2012). Fraud detection method called credible but used like an ‘instrument of medieval torture’. Science Magazine.

Hardcastle, M. J. (2015). The Effect of Selective Data Omission on Type I Error Rates: A Simulation Study (Doctoral dissertation, Texas A&M University).

Heino, M. The art of expecting p-values. Data Punk.

Simmons, J.P. , Nelson, L.D., &amp Simonsohn, U. (2014). P-Curve: A Key to the File-Drawer Journal of Experimental Psychology, 143(2), 534-537.

Simmons, J. P., & Simonsohn, U. (2017). Power posing: P-curving the evidence. Psychological Science.

Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2015). Better P-curves: Making P-curve analysis more robust to errors, fraud, and ambitious P-hacking, a Reply to Ulrich and Miller (2015).

Tanner, S. (2015). Evidence of false positives in research clearinghouses and influential journals: An application of P-curve to policy research. Observational Studies, 1, 18-29.


Replicability Index:

Replicability Index

Hughes, J. (2015). Evaluating the r-index and the p-curve. Disjointed Thinking.

Knutson, B. (2011). What scientific concept would improve everybody’s cognitive toolkit? Edge.

McCook, A. (2017). “I placed too much faith in underpowered studies:” Nobel Prize winner admits mistakes. Retraction Watch.

Schimmack, U., Heene, M., & Kesavan, K. (2017). Reconstruction of a train wreck: how priming research went off the rails. Replicability-Index.

Scudamore, C. L., Soilleux, E. J., Karp, N. A., Smith, K., Poulsom, R., Herrington, C. S., … & White, E. S. (2016). Recommendations for minimum information for publication of experimental pathology data: MINPEPA guidelines. The Journal of pathology, 238(2), 359-367.

A revised introduction to the r-index, (2016). Replicability-Index.

Replicability review of 2016, (2016). Replicability-Index.

Dr. R’s blog about replicability: top 10 list, (2016). Replicability-Index.

2016 replicability rankings of 103 psychology journals, (2017). Replicability-Index.

Hidden figures: replication failures in the stereotype threat literature, (2017). Replicability-Index.


Guidelines and Testing for Image Manipulation

Image Data Integrity – consulting services.

Abraham, E. (2008). The ATS Journals’ policy on image manipulation. ATS Journals, 5(9).

Cromey, D. (2013). Digital imaging: ethics. Southwest Environmental Health Sciences Center, University of Arizona.

Cromey, D.W. (2012).Digital images are data and should be treated as such. In Taatjes, D.J. & Roth, J. (Eds.), Cell imaging techniques: methods and protocols (pp. 1-27).

Cromey, D. W. (2010). Avoiding twisted pixels: ethical guidelines for the appropriate use and manipulation of scientific digital images. Science and engineering ethics, 16(4), 639-667.

Gel slicing and dicing: a recipe for disaster. (2004). Nature Cell Biology, 6(275).

Mudrak, B. Avoiding image fraud: 7 rules for editing images. American Journal Experts.

Martin, C., & Blatt, M. (2013). Manipulation and misconduct in the handling of image data. The Plant Cell, 25(9), 3147-3148.

Newman, A. (2013). The art of detecting data and image manipulation.

North, A.J. (2006). Seeing is believing? A beginners’ guide to practical pitfalls in image acquisition. The Journal of Cell Biology, 172(1), 9-18.

Rossner, M. & Yamada, K.M. (2004). What’s in a picture? The temptation of image manipulation. The Journal of Cell Biology, 166(1), 11-15.

Rossner, M. (2006). How to guard against image fraud. The Scientist.

Rossner, M. (2012). Digital images and misconduct. In The white paper on publication ethics (3.4). Council of Science Editors.


Preregistration and open data

AsPredicted: Preregistration.

AEA RCT Registry: The American Economic Association’s registry for randomized controlled trials.

Alvarez, R.M. (2014). The pros and cons of research preregistration. OUPblog.

COMET Initiative: Core Outcome Measures in Effectiveness Trials.

COMPARE: tracking switched outcomes in clinical trial

CONSORT: transparent reporting of trials.

EGAP: evidence in governance and politics. Preregistration.

Gelman, A. (2013). Preregistration of studies and mock reports. Political Analysis, 21(1), 40-41.

Gonzales, J. E., & Cunningham, C. A. (2015). The promise of pre-registration in psychological research. Psychological Science Agenda.

Lindsay, D. S., Simons, D. J., & Lilienfeld, S. O. (2016). Research preregistration 101. APS Observer, 29(10).

McCook, A. (2015). Did a clinical trial proceed as planned? New project finds out. Retraction Watch.

Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K.M., Gerber, A., …Van der Laan, M. (2014). Promoting transparency in social science research. Science, 343(6166), 30-31.

Monogan, J. (2014). The controversy of preregistration in social research. Berkeley Initiative for Transparency in the Social Sciences.

Nelson, L. (2014). Preregistration: not just for the empiro-zealots. Data Colada.

Neuroskeptic. (2014). Preregistration for data science? Discover.

NLM and NIH Clinical Trail Registry Site.

Nosek, B.A., Alter, G., Banks, G.C., Borsboom, D., Bowman, S.D., Breckler, S.J., … Yarkoni, T. (2015) Promoting an open research culture. Science, 348(6242), 1422-1425.

Open practice badges. Association for psychological science.

Open science: APS and it’s journals.

Open Science Framework.

Ozler, B. (2015). Preregistration of studies to avoid fishing and allow transparent discovery. Development Impact.

Pain, E. (2015). Register your study as a new publication option. Science.

Preregistration of research plans. Psychological Science.

Registrations: Center for Open Science.

Trust in science would be improved by study preregistration (2013). The Guardian.

van’t Veer, A. E., & Giner-Sorolla, R. (2016). Pre-registration in social psychology—A discussion and suggested template. Journal of Experimental Social Psychology, 67, 2-12.

The World Health Organization Registry: International Clinical Trails Registry Platform.


BPS invites readers to send (to krosnick@stanford.edu) relevant papers and links to add to this website.