Impact of post-publication notices on researcher’s trust: a randomized web-based survey

A significant part of the scientific literature is challenged in its ability to deliver valid information1,2. Leah et al. estimated that 14% of published studies reported false discoveries, which they defined as “cases where the null hypothesis is true in a hypothesis testing framework, but the results are reported as significant”3 . This can be caused by questionable research practices4, a high degree of flexibility in data collection and analysis5 or the presence of spin in the abstract and conclusion6, among other causes. End-point users use these studies to inform their practices. It is unclear whether they can detect problems with the quality of studies. Other studies have focused on the impact of spin on the perceived validity of research articles7–9. Spin refers to ‘specific reporting strategies, whatever their motive, to highlight that the experimental treatment is beneficial, despite a statistically non-significant difference of the primary outcome, or to distract the reader from statistically non-significant results.’, as defined by Boutron et al.6 They found that spin does influence medical professionals in their trust of the article. While the previously mentioned studies have studied the impact of poor scientific practices, they did not assess effective tools to prevent them from influencing endpoint users. A study evaluated the impact of including a “limitations” section to the abstract of systematic reviews on estimations of validity from reviewers, and found no significant difference10. This implies that including limitations is not enough to impact researcher’s trust in the findings. Other tools could be found in the form of post-publication notices or comments from other researchers, as they constitute a form of post-publication peer-reviews. The goal of this study is to assess the impact of post-publication notices on researchers' trust in scientific articles. Trust can be conceptualized as a truster’s subjective estimate of the probability that the trustee will display the truster’s preferred behavior11. In our case, it can be defined as a researcher’s (the truster) estimate of the probability that the abstract (the trustee) is giving accurate information (the preferred behaviour). Post-publication notices are notices issued by a journal to notify readers of an article of issues related to its content. Corrections, expressions of concern and retractions are considered post-publication notices. Additionally, we chose to include specific PubPeer - a collaborative platform that allows users to discuss published articles - comments alongside these notices, as they can play a significant role in signaling potential issues. Our primary hypothesis is that the presence of a post-publication notice or a PubPeer comment decreases trust in the given information. In this study, participants will be asked to complete a survey. They will be presented with abstracts of real, published RCTs12–14 with or without their respective attached post-publication notice. The survey will examine 1) the impact of post-publication notices on the perceived effectiveness of the intervention discussed in the abstract; 2) its impact on the perceived quality of the methodology; 3) its impact on how many end-users will want to continue reading past the abstract. 1. Ioannidis, J. P. A. Why Most Published Research Findings Are False. PLoS Med. 2, e124 (2005). 2. Munafò, M. R. et al. A manifesto for reproducible science. Nat. Hum. Behav. 1, 0021 (2017). 3. Jager, L. R. & Leek, J. T. An estimate of the science-wise false discovery rate and application to the top medical literature. Biostatistics 15, 1–12 (2014). 4. John, L. K., Loewenstein, G. & Prelec, D. Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling. Psychol. Sci. 23, 524–532 (2012). 5. Simmons, J. P., Nelson, L. D. & Simonsohn, U. False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychol. Sci. 22, 1359–1366 (2011). 6. Boutron, I. Reporting and Interpretation of Randomized Controlled Trials With Statistically Nonsignificant Results for Primary Outcomes. JAMA 303, 2058 (2010). 7. Boutron, I. et al. Impact of Spin in the Abstracts of Articles Reporting Results of Randomized Controlled Trials in the Field of Cancer: The SPIIN Randomized Controlled Trial. J. Clin. Oncol. 32, 4120–4126 (2014). 8. Khanpara, H. & Prakash, V. Effect of spin in the abstract of a randomised controlled trial on physiotherapists’ perception of treatment benefit: a randomised controlled trial. BMJ Evid.-Based Med. 27, 97–103 (2022). 9. Shinohara, K. et al. Influence of overstated abstract conclusions on clinicians: a web-based randomised controlled trial. BMJ Open 7, e018355 (2017). 10. Yavchitz, A., Ravaud, P., Hopewell, S., Baron, G. & Boutron, I. Impact of adding a limitations section to abstracts of systematic reviews on readers’ interpretation: a randomized controlled trial. BMC Med. Res. Methodol. 14, 123 (2014). 11. Bauer, P. Conceptualizing Trust and Trustworthiness. Res. Gate (2019). 12. Logunov, D. Y. et al. Safety and efficacy of an rAd26 and rAd5 vector-based heterologous prime-boost COVID-19 vaccine: an interim analysis of a randomised controlled phase 3 trial in Russia. The Lancet 397, 671–681 (2021). 13. Xu, L.-L. et al. RETRACTED: Efficacy and Safety of Esketamine for Supplemental Analgesia During Elective Cesarean Delivery: A Randomized Clinical Trial. JAMA Netw. Open 6, e239321 (2023). 14. Keller, M. B. et al. Efficacy of Paroxetine in the Treatment of Adolescent Major Depression: A Randomized, Controlled Trial. J. Am. Acad. Child Adolesc. Psychiatry 40, 762–772 (2001).