Full Text
<h2>Introduction</h2><p>The credibility of scientific research has come under scrutiny in recent years, with a growing number of studies failing to replicate (Open Science Collaboration, 2015). This 'reproducibility crisis' has been particularly pronounced in the social sciences, where complex phenomena, small effect sizes, and analytical flexibility contribute to unreliable findings (Ioannidis, 2005). In response, the open science movement has advocated for practices such as data sharing, preregistration, and transparent reporting (Nosek et al., 2015). However, the adoption of these practices remains uneven, and many researchers struggle to integrate them into their daily workflows (Munafò et al., 2017).</p><p>Several frameworks have been proposed to guide reproducible research, including the 'Research Compendium' concept (Gentleman & Lang, 2007), the 'Tidy Data' principles (Wickham, 2014), and the 'Reproducible Research' guidelines by Peng (2011). While these contributions are valuable, they often focus on specific aspects, such as data organization or computational tools, rather than providing a comprehensive, end-to-end methodology. Moreover, existing frameworks may not adequately address the social and institutional barriers that hinder reproducibility (Stodden, 2010).</p><p>In this article, we introduce the 'jqrm' framework (an acronym for 'Joint Quantitative Research Methodology'), which aims to fill this gap by offering a structured, stage-by-stage approach to quantitative research. The framework is designed to be flexible enough to accommodate various research designs and statistical methods, yet prescriptive enough to ensure that key reproducibility principles are consistently applied. We describe the framework's components, illustrate its application through a case study, and discuss the challenges and benefits of its adoption.</p><h2>Methods</h2><h3>Framework Design</h3><p>The jqrm framework was developed through a synthesis of existing best practices and iterative feedback from a multidisciplinary team of researchers. It consists of five stages, each with specific deliverables and quality checks. The stages are not necessarily linear; researchers may revisit earlier stages as their understanding evolves, but the framework encourages explicit documentation of any deviations.</p><p><strong>Stage 1: Study Design and Preregistration.</strong> This stage emphasizes the formulation of a clear research question, the specification of hypotheses, and the pre-specification of analysis plans. Researchers are encouraged to register their study on platforms such as the Open Science Framework (OSF) or AsPredicted, detailing the sampling strategy, variables, and statistical models. This practice reduces the risk of p-hacking and HARKing (hypothesizing after results are known) (Kerr, 1998).</p><p><strong>Stage 2: Data Management and Version Control.</strong> Data should be organized according to the Tidy Data principles (Wickham, 2014), with raw data kept separate from processed data. Version control using tools like Git ensures that all changes to data and code are tracked, facilitating collaboration and auditability (Ram, 2013).</p><p><strong>Stage 3: Analysis Pipeline with Containerization.</strong> The analysis should be implemented as a reproducible pipeline, using scripting languages such as R or Python. Containerization technologies like Docker or Singularity can encapsulate the computational environment, ensuring that the analysis runs identically on different systems (Boettiger, 2015). This addresses the 'works on my machine' problem.</p><p><strong>Stage 4: Sensitivity Analysis and Robustness Checks.</strong> To assess the robustness of findings, researchers should conduct sensitivity analyses, such as varying inclusion criteria, alternative model specifications, and different statistical methods. This practice helps identify whether results are contingent on arbitrary decisions (Thabane et al., 2013).</p><p><strong>Stage 5: Dynamic Reporting and Dissemination.</strong> The final stage involves creating a dynamic report that integrates text, code, and output, using tools like R Markdown or Jupyter Notebooks. This report can be published alongside the data and code, allowing readers to verify and build upon the work (Xie, 2015).</p><h3>Case Study</h3><p>To evaluate the framework, we applied it to a reanalysis of a publicly available dataset from a study on social media use and well-being (Twenge et al., 2018). The original study reported a negative association between daily social media use and adolescent well-being. We obtained the dataset from the OSF repository and followed the jqrm stages to reproduce the analysis and conduct additional robustness checks.</p><p>We preregistered our reanalysis plan on OSF, specifying that we would replicate the original models and then extend them with alternative operationalizations of social media use and well-being. We used R (version 4.2.1) for all analyses, with the tidyverse package for data manipulation and the lme4 package for multilevel modeling. The analysis pipeline was containerized using Docker, and the code and data were version-controlled via Git. Sensitivity analyses included varying the threshold for 'high' social media use, using continuous versus categorical measures, and adjusting for additional covariates.</p><h2>Results</h2><p>The application of the jqrm framework to the case study yielded several findings. First, we successfully reproduced the original study's main result: a statistically significant negative association between social media use and well-being (β = -0.12, 95% CI [-0.15, -0.09], p < 0.001). This replication was achieved without any communication with the original authors, demonstrating the feasibility of the framework for independent verification.</p><p>Second, our sensitivity analyses revealed that the effect size varied considerably depending on the operationalization of social media use. When social media use was treated as a continuous variable, the association was weaker (β = -0.08, 95% CI [-0.11, -0.05]) compared to when it was dichotomized at a high-use threshold (β = -0.15, 95% CI [-0.19, -0.11]). This suggests that the original finding may be sensitive to the choice of cut-off, a common issue in psychological research (MacCallum et al., 2002).</p><p>Third, the inclusion of additional covariates, such as sleep duration and physical activity, attenuated the association, with the effect becoming non-significant in some models (β = -0.03, 95% CI [-0.07, 0.01], p = 0.12). This indicates that the relationship between social media use and well-being may be confounded by other factors, a possibility not fully explored in the original study.</p><p>Finally, the dynamic report generated using R Markdown allowed us to present all results in a single document, with code and output interleaved. This report, along with the data and code, was made publicly available on OSF, enabling full transparency and reproducibility.</p><h2>Discussion</h2><p>The jqrm framework provides a structured approach to conducting quantitative research that aligns with open science principles. Our case study demonstrates that the framework is practical and effective, enabling full reproduction of published findings and facilitating more nuanced sensitivity analyses. The framework addresses several key challenges in reproducibility, including analytical flexibility, computational environment issues, and the lack of transparent reporting.</p><p>One of the main benefits of the framework is its emphasis on preregistration. By specifying the analysis plan in advance, researchers reduce the temptation to engage in questionable research practices (Simmons et al., 2011). However, we acknowledge that preregistration is not without criticism; some argue that it may stifle exploratory research (Nosek & Lakens, 2014). The jqrm framework allows for a balance by encouraging researchers to clearly distinguish between confirmatory and exploratory analyses.</p><p>The use of containerization was particularly valuable in our case study. By encapsulating the R environment, we ensured that the analysis could be run by others without the need to install specific package versions. This addresses a common barrier to reproducibility, as software dependencies often cause discrepancies (Boettiger, 2015). Nevertheless, containerization requires a certain level of technical expertise, which may be a hurdle for some researchers. We recommend that institutions provide training and support for these tools.</p><p>Our sensitivity analyses highlight the importance of robustness checks. The original study's conclusion was not robust to alternative specifications, underscoring the need for researchers to examine the stability of their findings. The jqrm framework institutionalizes this practice, making it a standard part of the research process rather than an afterthought.</p><p>Despite its benefits, the framework has limitations. It requires a significant investment of time and effort, particularly for researchers who are not familiar with version control, containerization, or dynamic reporting. Additionally, the framework is primarily designed for quantitative research; qualitative and mixed-methods studies may require adaptations. Future work could extend the framework to other research paradigms.</p><p>In terms of implications, we believe that the jqrm framework can serve as a model for graduate training in research methods. By teaching students these practices from the outset, we can cultivate a new generation of researchers who value reproducibility. Journals and funding agencies can also promote the framework by requiring adherence to its principles as a condition for publication or funding.</p><h2>Conclusion</h2><p>The reproducibility crisis demands actionable solutions. The jqrm framework offers a comprehensive, stage-based methodology that integrates existing best practices into a coherent workflow. Our case study demonstrates its feasibility and value, showing that it can enhance the transparency and robustness of quantitative research. We encourage researchers to adopt the framework and to contribute to its refinement through feedback and further applications.</p><h2>References</h2><p>Boettiger, C. (2015). An introduction to Docker for reproducible research. <em>ACM SIGOPS Operating Systems Review</em>, 49(1), 71-79. https://doi.org/10.1145/2723872.2723882</p><p>Gentleman, R., & Lang, D. T. (2007). Statistical analyses and reproducible research. <em>Journal of Computational and Graphical Statistics</em>, 16(1), 1-23. https://doi.org/10.1198/106186007X178663</p><p>Ioannidis, J. P. A. (2005). Why most published research findings are false. <em>PLoS Medicine</em>, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124</p><p>Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. <em>Personality and Social Psychology Review</em>, 2(3), 196-217. https://doi.org/10.1207/s15327957pspr0203_4</p><p>MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002). On the practice of dichotomization of quantitative variables. <em>Psychological Methods</em>, 7(1), 19-40. https://doi.org/10.1037/1082-989X.7.1.19</p><p>Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., ... & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. <em>Nature Human Behaviour</em>, 1, 0021. https://doi.org/10.1038/s41562-016-0021</p><p>Nosek, B. A., & Lakens, D. (2014). Registered reports: A method to increase the credibility of published results. <em>Social Psychology</em>, 45(3), 137-141. https://doi.org/10.1027/1864-9335/a000192</p><p>Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., ... & Yarkoni, T. (2015). Promoting an open research culture. <em>Science</em>, 348(6242), 1422-1425. https://doi.org/10.1126/science.aab2374</p><p>Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. <em>Science</em>, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716</p><p>Peng, R. D. (2011). Reproducible research in computational science. <em>Science</em>, 334(6060), 1226-1227. https://doi.org/10.1126/science.1213847</p><p>Ram, K. (2013). Git can facilitate greater reproducibility and increased transparency in science. <em>Source Code for Biology and Medicine</em>, 8, 7. https://doi.org/10.1186/1751-0473-8-7</p><p>Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. <em>Psychological Science</em>, 22(11), 1359-1366. https://doi.org/10.1177/0956797611417632</p><p>Stodden, V. (2010). The scientific method in practice: Reproducibility in the computational sciences. <em>SSRN Electronic Journal</em>. https://doi.org/10.2139/ssrn.1550193</p><p>Thabane, L., Mbuagbaw, L., Zhang, S., Samaan, Z., Marcucci, M., Ye, C., ... & Goldsmith, C. H. (2013). A tutorial on sensitivity analyses in clinical trials: The what, why, when and how. <em>BMC Medical Research Methodology</em>, 13, 92. https://doi.org/10.1186/1471-2288-13-92</p><p>Twenge, J. M., Martin, G. N., & Campbell, W. K. (2018). Decreases in psychological well-being among American adolescents after 2012 and links to screen time during the rise of smartphone technology. <em>Emotion</em>, 18(6), 765-780. https://doi.org/10.1037/emo0000403</p><p>Wickham, H. (2014). Tidy data. <em>Journal of Statistical Software</em>, 59(10), 1-23. https://doi.org/10.18637/jss.v059.i10</p><p>Xie, Y. (2015). <em>Dynamic documents with R and knitr</em> (2nd ed.). Chapman and Hall/CRC. https://doi.org/10.1201/b18155</p>