Selective Reporting: Outcomes, Experimental Conditions, and Studies
General Information
Selective reporting or ‘reporting bias’ determines which studies ‘come to light’ and which do not (Glen, 2016). There are many types of ‘selective reporting’ such as citation bias, language bias, location bias, duplicate publication bias, outcome reporting bias, publication bias, and time lag bias. One can see full definitions of these biases by clicking here. Selective reporting is not just the published versus unpublished spectrum, but “a continuum ranging from the sharing of draft papers among colleagues, through presentations at meetings and published abstracts, to papers in journals that are indexed in the major bibliographic databases” (Cochrane Methods Bias).
There is also another type of selective reporting called, ‘selective outcome reporting.’ This occurs “wherein only a subset of the original outcomes measured and analyzed in a study are fully reported based on the magnitude of the treatment effect or the statistical significance of selected outcomes” (Norris SL, 2012). There are three types of selective outcome reporting (SOR): selective reporting of an entire study outcome, selective reporting of a specific outcome, and incomplete reporting of a specific outcome.
In addition, selective outcome reporting can occur in many different ways. According to Henegan, selective outcome reporting can appear from omitting outcomes which are deemed to be unfavorable or statistically insignificant, adding new outcomes based on collected data to favor statistical significance, including only a subset of the analyzed data in the published study, failing to report data that was analyzed in the trial (such as adverse effects), and changing outcomes of interest (from primary outcomes to secondary outcomes if they do not yield significant results) (2017). You can read more about selective reporting and selective outcome reporting below.
Here are resources on the phenomenon:
Glen, S. (2016). Reporting bias: definition and examples, types. Statistics How To.
Andale (2013). Bias in statistics: definition, selection bias & survivorship bias. Statistics How To.
Bachet, J. & Morduch, J. (2009). Selective knowledge: reporting biases in microfinance data. The Financial Access Initiative.
Butler, N., Delaney, H., & Spoelstra, S. (2017). The Gray Zone: Questionable Research Practices in the Business School. Academy of Management Learning & Education, 16(1), 94-109.
Custer, R. (2013). Research misconduct – the grey area of questionable research practices. VIB.
Dickersin, K., & MIN, Y. I. (1993). Publication bias: the problem that won’t go away. Annals of the New York Academy of Sciences, 703(1), 135-148.
Franco, A., Malhotra, N., & Simonovits, G. (2015). Underreporting in Political Science Survey Experiments: Comparing Questionnaires to Published Results.Political Analysis, 23(2).
Frontiers (2017). Exacerbating the replication crisis in science: replication studies are often unwelcome. PHYS.ORG.
Henegan, C. (2017). Outcome reporting bias. Catalogue of Bias.
Higgins, J., Altman, D.G., & Sterne, J. (Eds.) (2011). Chapter 8: assessing risk of bias in included studies. In Higgins, J. & Green, S. (Eds.), The Cochrane handbook for systematic reviews of interventions.
John, L.K. (2012). Questionable research practices surprisingly common. Association for Psychological Science.
Mahtani, K. Outcome reporting bias: if you say you’re going to do something, do it! Centre For Evidence-Based Medicine.
Mahtani, K. Outcome reporting bias: is it ok to be a little selective? Centre for Evidence-Based Medicine.
Møller, A. P., & Jennions, M. D. (2001). Testing and adjusting for publication bias. Trends in Ecology & Evolution, 16(10), 580-586.
Norris, S.L., Holmer, H.K., Ogden, L.A., Fu, R.F., Abou-Setta, A.M., Viswanathan, M.S., McPheeters, M.L. (2012). Methods research report: selective outcome reporting as a source of bias in reviews of comparative effectiveness. Agency for Healthcare Research and Quality, Publication No. 12-EHC110-EF.
Norris S.L., H, et. al. (2012). Selective Outcome Reporting as a Source of Bias in Reviews of Comparative Effectiveness. NCBI.
Outcome Reporting Bias in Trials (ORBIT).
Page, M. J., McKenzie, J. E., Kirkham, J., Dwan, K., Kramer, S., Green, S., & Forbes, A. (2014). Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions. The Cochrane Library.
Questionable research practices: definition, detect, and recommendations for better practices (2015). Replicability-Index.
Rennie, D., & Flanagin, A. (1992). Publication bias: the triumph of hope over experience. Jama, 267(3), 411-412.
Reporting biases. Cochrane Methods Bias.
Salandra, R. (2015). Selective reporting in industrial research: the effect of innovation, uncertainty of science and competition on firm motivation. Druid academy.
Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods, 17(4), 551-566.
Simmons, J.P., Nelson, L.D., Simonsohn, U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science.
Sterne, J., Egger, M., & Moher, D. (Eds.) (2011). Chapter 10: addressing reporting biases. In Higgins, J. & Green, S. (Eds.), The Cochrane handbook for systematic reviews of interventions.
Thornton, A., & Lee, P. (2000). Publication bias in meta-analysis: its causes and consequences. Journal of clinical epidemiology, 53(2), 207-216.