Download to read offline
Uploaded bySusanna-Assunta Sansone
FAIR and Reproducible - GSC, Tucson, Aug 2024
The document outlines challenges and strategies in achieving reproducibility in genomics research, emphasizing that cultural change is essential for its success. It discusses the importance of FAIR principles and the need for common guidelines among communities and stakeholders to foster data management practices. Additionally, the document highlights ongoing initiatives, such as the creation of an editorial reference handbook, aimed at operationalizing checks for reproducible research within journals.
More Related Content
Similar to FAIR and Reproducible - GSC, Tucson, Aug 2024
FAIR and Reproducible - GSC, Tucson, Aug 2024
- 1. From FAIR toReproducible: spotlight on TIER2 GSC24 Challenges of Reproducibility in Genomics, Bio5, University of Arizona, Tucson, 6-9 August 2024 https://www.slideshare.net/SusannaSansone Susanna-Assunta Sansone, PhD 0000-0001-5306-5690 @SusannaASansone [email protected] datareadiness.eng.ox.ac.uk sasansone Professor of Data Readiness Director of the Oxford e-Research Centre Academic Lead for Research Practice
- 2. To enhance thevalue of all digital resources and its reuse by humans and machines To define a continuum of increasing reusability, via many different implementations To align many communities and stakeholders around common guidelines A global norm for good data management https://doi.org/10.1038/sdata.2016.18
- 3. FAIR and OSpractices underpin reproducibility https://www.knowledge-exchange.info/event/fair-data-and-software 2024 https://doi.org/10.5281/zenodo.10663902 2024 https://doi.org/10.5281/zenodo.10664659 2022 https://theplosblog.plos.org/2022/07/reproducibility
- 4. FAIR and OSpractices underpin reproducibility https://www.knowledge-exchange.info/event/fair-data-and-software 2024 https://doi.org/10.5281/zenodo.10663902 2024 https://doi.org/10.5281/zenodo.10664659 ● What else is required to move reproducible research from innovative exceptions to a commonly accepted way of doing science? ● The challenges go far beyond technology and principles: ○ it is a cultural change! 2022 https://theplosblog.plos.org/2022/07/reproducibility
- 5. “Hey, nice pedestal,great job!” https://blog.x.company/tackle-the-monkey-first-90fd6223e04d Source of the concept: Google’s in- house innovation hub Credit: to Tony Ross-Hellauer for the idea!
- 6.
- 7. Communities Communities Communities Communities Communities Communities Incentives Incentives Incentives Infrastructure and Skills Infrastructureand Skills Infrastructure and Skills Infrastructure and Skills Infrastructure and Skills Infrastructure and Skills Usability Usability Usability Usability Usability Usability Policy Awareness and Understandability The road to FAIRness and reproducibility Modified form the Theory of Change model (Nosek, 2019): https://www.cos.io/blog/continuing-acceleration-new-strategic- plan and https://zenodo.org/record/6881009#.Y2BIeuTP2F5 D4.4 Report and recommendations on FAIR incentives and
- 8. https://doi.org/10.5281/zenodo.5521077 How far alongare we? “The reproducibility discourse is less developed than these comparators – not least because it largely relies on a culture of open science that is not yet fully developed.” Comparison between the policy landscape:
- 9. Scope: • All fieldsof science • Survey of researchers: 15066 responses • Survey of research data repositories: 316 responses • Desk research; case studies; FAIRness assessment 2022 https://data.europa.eu/doi/10.2777/3648 Objectives: • To collect data on the level of maturity with respect to FAIR data implementation • To assess responsiveness and readiness of research data repositories in terms of implementation of FAIR principles Is FAIR delivering on its promises?
- 10. Globally unique and persistentidentifiers Community defined descriptive metadata Community defined terminologies Detailed provenance Terms of access Terms of use A set of principles…..not a standard! The narrative is insufficient to circumscribe the valid mechanism to achieve the behaviours they describe Tendency towards “gamification” of FAIR-compliance, with little/no proof they are or support FAIRness What is the problem?
- 11. As of Aug2024, there are 28 independent evaluation, assessment, assistance tools; see fairassist.org The tests used and the results given are inconsistent, and not comparable The “cottage industry” of FAIRness We need to trust claims such as “I am FAIR”, or “My data is FAIR”, or “My repository enables FAIR”
- 12. European funding programme:focus on FAIR
- 13. European funding programme:over 24 projects https://eosc.eu/horizon-europe-projects 24 (ongoing) EOSC projects, led by infrastructure developers and service providers
- 14. OSTrails: addressing thedefinition of FAIRness https://eosc.eu/horizon-europe-projects zenodo.org/doi/10.5281/zenodo.10490288 zenodo.org/record/7463421 in collaboration with Are working to harmonise how FAIR evaluation, assessment and assistance are provided 24 (ongoing) EOSC projects, led by infrastructure developers and service providers
- 15. https://eosc.eu/horizon-europe-projects https://www.irise-project.eu https://tier2-project.eu https://osiris4r.eu 3 (ongoing) projectson reproducibility, led by specific disciplines like medicine & psychology 24 (ongoing) EOSC projects, led by infrastructure developers and service providers What about projects focussing on reproducibility?
- 16. https://www.irise-project.eu https://tier2-project.eu https://osiris4r.eu What about projectsfocussing on reproducibility? Know-Center GmbH – Project Coordinator Athena Research Center Biomedical Research Center Fleming Stichting VUmc Amsterdam Aarhus University Pensoft Publishers Gesis Leibniz Institute for Social Sciences OpenAIRE Charité - Universitätsmedizin Berlin University of Oxford https://tier2-project.eu
- 17. ● Ongoing challengeswith the sharing of datasets, code, material and other digital objects underpinning results in publications Reproducibility and the publishers 2023 https://doi.org/10.17605/OSF.IO/TGUXZ
- 18. ● Ongoing challengeswith the sharing of datasets, code, material and other digital objects underpinning results in publications ● Consensus to provide education and guidance to the journal (in- house) teams on how to: ○ operationalise a practical set of checks, contributing towards a common understanding and what is required to ensure digital objects are shared in a FAIR manner and ultimately reproducible; ○ integrate the checks in journal policies and in the manuscript submission workflow Reproducibility and the publishers 2023 https://doi.org/10.17605/OSF.IO/TGUXZ
- 19. ● Ongoing challengeswith the sharing of datasets, code, material and other digital objects underpinning results in publications ● Consensus to provide education and guidance to the journal (in- house) teams on how to: ○ operationalise a practical set of checks, contributing towards a common understanding and what is required to ensure digital objects are shared in a FAIR manner and ultimately reproducible; ○ integrate the checks in journal policies and in the manuscript submission workflow ● Key requirements: ○ a small number of checks, feasible and achievable ○ an incremental alignment and consistency across journals and publishers ○ not just the pedestal…. Reproducibility and the publishers 2023 https://doi.org/10.17605/OSF.IO/TGUXZ
- 20. Views and opinionsexpressed are those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Executive Agency (REA). Neither the EU nor REA can be held responsible for them. Pilot: Editorial Reference Handbook Co-Leads: Allyson Lister, Susanna-Assunta Sansone (University of Oxford) Rebecca Taylor-Grant, Matt Cannon (Taylor & Francis) Intervention lead: Christopher Osborne (University of Oxford)
- 21. Editorial Reference Handbook •Co-creation of the checks and guidance to operationalise them • The handbook aims also to put the requirements of the journal data policy in action • Journals that already have their own internal guidance will be able to use the handbook to validate and refine their existing methodology • Journals that do not yet have their own internal guidance should use it as an opportunity to define their own process
- 22. Editorial Reference Handbook •Co-creation of the checks and guidance to operationalise them • The handbook aims also to put the requirements of the journal data policy in action • Journals that already have their own internal guidance will be able to use the handbook to validate and refine their existing methodology • Journals that do not yet have their own internal guidance should use it as an opportunity to define their own process • Intervention to get it tested/implemented by in-house editorial staff managing the manuscripts • It will also benefit reviewers, authors and service providers by making the checks transparent and understandable to them
- 23. Co-creation phase, part1: the checks • An educational and practical set of checks in support of reproducibility and FAIRness Reviewed existing guidance for reproducibility/FAIRness checks (MDAR, PRO MaP, FAIR4RS, F1000 guidelines, Nature editorial checklist… Created list of 12 core checks: From FAIRsharing, we create reports to assist with policy auditing
- 24. Co-creation phase, part2: the internal process • A general framework to help improve internal processes, where needed • There is a variety of internal processes, and how, when and by whom these checks are done vary, and this can also affect the results
- 25.
- 26. Credit to ChristopherOsborne, PhD student, University of Oxford https://doi.org/10.1186/1748-5908-6-42 Intervention: the behaviour change wheel • An evidence-based intervention to increase the implementation of the Handbook
- 27. The world isfull of pedestals …. https://doi.org/10.1371/journal.pbio.3001943