by Marjorie Mitchell
Research Librarian, UBC Okanagan Library
I recently learned about the concept of researcher degrees of freedom. The idea is that, as researchers, we make many decisions about the data we collect and how we analyze it (Simmons, Nelson & Simonsohn, 2011) and that these decisions have a direct impact on the results of our research. Obviously. And this is what begins to make this concept so powerful. I’ve been familiar with the idea of researcher bias, but this is a more subtle concept. I want to make it clear that I am not talking about deliberate falsifying of research results, but about the real choices a researcher who has gathered data, in any form, is faced with making during the journey from research question construction through gathering data through publication. Some of these choices include what data is being gathered and what is not, how to handle data outliers, how to group or cluster points of data when looking for significance or impact, and so on.
The concept has been around for some time and has been explored in the fields of medicine, psychology, and more widely in the sciences, where researchers were finding they could not replicate the results from published research studies. It’s spawned a whole area of research on false positive results, publication bias (more papers are published that show positive results than show negative results), selective reporting of results, and an “Open Science” network.
Evidence-based practice, no matter what field or discipline, has been one method of critically analyzing research results. It’s a valuable tool, but only one of many that can be applied. I, personally, would like to see more credit being given for rigorously trying to replicate previously conducted research. I would also like to see the publication of more null-results reports. It is incredibly handy to know a particular path is not a useful path to follow. I have to admit, I don’t know whether I’m ready to participate in the registration of my research methods in advance of collecting data. “A registration is useful for certifying what you did in a project in advance of data analysis, or for confirming the exact state of the project at important points of the lifecycle, such as manuscript submission or the onset of data collection” (Centre for Open Science, 2018). This certainly raises the bar for research.
We have a journal club in my library where we meet to discuss recent articles from the library literature. I am pleased with the “skeptical eye” we often apply to the articles we read, but I wonder whether our critical and skeptical reading ultimately makes us less biased researchers.
More often than not, it is challenging to attempt to make these decisions in advance of conducting out research. I admit to thinking “I’ll just have a look at the results of my survey before I commit to an hypothesis.” Often my research never reaches the publication stage but our research will ultimately be stronger if we try to make these decisions in advance.
I continue to look for new lenses to apply to the research I do and to the research I consume. It has not been my intent to criticize the people doing research but to discuss the challenges and struggles I wrestle with as I continue down a research path. I hope this concept will be of interest to others as well.
References
Center for Open Science. (2018).OSF FAQs What is registration? https://cos.io/our-services/top-guidelines/
Dickersin, K., Chan, S., Chalmersx, T. C., Sacks, H. S., & Smith, H. (1987). Publication bias and clinical trials. Controlled Clinical Trials, 8(4), 343-353. doi:10.1016/0197-2456(87)90155-3
Kepes, S., Banks, G. C., & Oh, I. (2014). Avoiding bias in publication bias research: The value of “null” findings. Journal of Business and Psychology, 29(2), 183-203. doi:10.1007/s10869-012-9279-0
Pickett, J. T., & Roche, S. P. (2018). Questionable, objectionable or criminal? public opinion on data fraud and selective reporting in science. Science and Engineering Ethics, 24(1), 151-171.
doi:10.1007/s11948-017-9886-2
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359-1366. doi:10.1177/0956797611417632
Stahl, D., & Pickles, A. (2018). Fact or fiction: Reducing the proportion and impact of false positives. Psychological Medicine, 48(7), 1084. doi:10.1017/S003329171700294X
Pickett, J. T., & Roche, S. P. (2018). Questionable, objectionable or criminal? public opinion on data fraud and selective reporting in science. Science and Engineering Ethics, 24(1), 151-171. doi:10.1007/s
This article gives the views of the author(s) and not necessarily the views of the Centre for Evidence Based Library and Information Practice or the University Library, University of Saskatchewan.