Web Science and Digital Libraries Research Group

         The “reproducibility crisis” in scientific research refers to growing concerns over the reliability and credibility of published findings, in many fields including biomedical, behavioral, and the life sciences ( Laraway et al. 2019 , Fidler et al. 2021 ). Over the past decade, large-scale reproducibility projects revealed failures to replicate findings. For example, in 2015 the Open Science Collaboration reported that a larger portion of replicated studies produced weaker evidence for the original findings despite using the same materials. Similarly, in fields like machine learning researchers may publish impressive new methods, but if others can’t reproduce the results, it hinders progress. Therefore, reproducibility matters. It’s science’s version of fact-checking. In this blog, we’ll break down our recent effort to reproduce the results of the paper Network Deconvolution by Ye et al. (hereafter, "original study"), published in 2020, which claime...

I am excited to announce that Old Dominion University (ODU) is part of the multi-university grant awarded by the Open Philanthropy Foundation  to support the development of a systematic benchmark assessing how effectively large language models (LLMs) can evaluate, replicate, and conduct scientific research. The leading institution is the Center for Open Science (CoS, Dr. Brian Nosek and Dr. Tim Errington ) and the participation institutions are Pennsylvania State University ( Dr. Sarah Rajtmajer , Dr. Qingyun Wu ), Notre Dame University ( Dr. Meng Jiang ), and ODU ( Dr. Jian Wu , myself).  The team will conduct a test of whether LLMs are capable of determining whether claims are true or false. Here a claim means a statement that conveys a research finding in a scientific paper. Our operationalization of this question is whether LLMs can assess a scientific paper and predict whether primary findings would replicate or reproduce successfully in an independent test. In the fund...

    Figure 1: Homepage of ACM REP ‘24 The ACM Conference for Reproducibility and Replicability (ACM REP) is a recently launched venue for computer and information scientists whose research covers reproducibility and replicability. It covers practical issues in computational science and data exploration, emphasizing the importance of community collaboration to adopt new methods and tackle challenges in this relatively young field. The conference is linked with the ACM Emerging Interest Group for Reproducibility and Replicability . The 2024 ACM Conference on Reproducibility and Replicability (ACM REP ‘24) combined in-person and virtual participation to broaden participation. Held from June 18th to June 20th at the Inria Conference Center in Rennes, France, the conference featured tutorials, keynote speeches, paper presentations, and discussions on reproducibility in computational research. The conference had four tutorial sessions; each divided into three to four tracks, s...