Berlin 6 Open Access Conference: Jelena Kovacevic
Download as PPT, PDF1 like766 views
This document discusses issues with reproducibility in research and proposes some solutions: 1. Reproducible research is important for computational sciences but faces challenges from cultural norms that prioritize innovation over reproducibility and from intellectual property concerns. 2. To encourage more reproducible research, journals could establish special sections, paper awards, or guidelines for what reproducible research papers should contain. Researchers should also be encouraged to freeze code upon publication and make data and algorithms transparent. 3. A case study of 15 papers found that while proofs were provided, no papers made their code or full data available, highlighting the need for more reproducible practices.
Berlin 6 Open Access Conference: Jelena Kovacevic
- 1. How to Encourage and Publish Reproducible Research Jelena Kovačević Center for Bioimage Informatics Department of Biomedical Engineering Department of Electrical & Computer Engineering Carnegie Mellon University
- 2. Theory Vs Experimentation Theoretical disciplines Mathematics Experimental disciplines Biology A hybrid is born: Computational sciences Should follow good practices from both SP falls in there: How are we doing? Axioms Lemmas Proof MIT CMU Stanford Harvard hypothesis
- 3. Issues Cultural Innovation above all else TIP Transactions reviewing questions 1. Is the paper technically sound? 2. Is the coverage of the topic sufficiently comprehensive and balanced? 3. How would you describe the technical depth of the paper? 4. How would you rate the technical novelty of the paper? Can lead to paradox Educational Our students undertrained in statistics Typically reimplement everything Data We collaborate and data might not be ours IP Data issues Companies and agencies protecting their IP Collaborative With colleagues within the university/company, outside
- 4. How Do We Publish RR? Not likely to happen overnight Encourage and reward “good behavior” (Child psychology 101) Ideas Special section in Transactions for RR? Establish a paper award for an RR paper? Form a rough guideline of what each paper should contain for an RR designation? Everything we read is partly “on faith”
- 5. How to Make Papers RR? Used in my group Compilation of ideas from Barni and EPFL groups (Vetterli, Vandewalle et al.) Compendium (Gentleman & Lang) Freeze the code upon Submission Acceptance “ Good intentions” enforced Students do projects and reproduce
- 6. An Entirely NonRR Case Study Data set 15 papers published in the TIP EDICS category using both theory and experimentation Stayed away from standards as well as biomed For all algorithms, competing ones exist RR block-diagram? pseudo code? data available? code available? proof available? Ratings (0, 0.5, 1) Algorithm and experimental setup algorithm explained? data explained? data size? details on parameters used? comparison to competing algorithms?
- 7. Results of the Entirely NonRR Case Study All papers had proofs, none had code available Sufficient detail on algorithms, none had a block-diagram Data used, data size and availability all below average Half of the cases were the parameters specified Comparisons to competing algorithms: quarter Pleasant surprise: 60%, pseudo-code was available
Editor's Notes
- #2: Jelena Kovačević: MR Tools for Next-Generation Bioinformatics Systems 06/06/09 This is a brief version of the presentation I gave at Icassp 2007 in the special session on reproducible research. My goal was to give some input based on my experience during the 5 years I served as the Editor-in-Chief of the IEEE Transactions on Image Processing.