SlideShare a Scribd company logo
Reproducibility in cheminformatics and
computational chemistry research: Certainly
we can do better than this
Gregory Landrum Ph.D.
NIBR IT
Novartis Institutes for BioMedical Research
Basel
GCC 2012 Goslar
Outline
§  Reproducibility?
§  Requirements for reproducibility of published research
§  Practical aspects
Landrum, G. A. & Stiefl, N. Is that a scientific publication or an advertisement?
Reproducibility, source code and data in the computational chemistry literature. Future
Medicinal Chemistry 4, 1885–1887 (2012).
But first!
A new fingerprint for similarity-based virtual
screening
§  Start with Morgan fingerprints (a.k.a. circular fingerprints1)
§  The usual FCFP algorithm uses fairly crude feature definitions
§  Combine the RDKit Morgan fingerprint algorithm with pharmacophoric
features calculated using “better” feature definitions2.
1.  Rogers, D. & Hahn, M. Extended-Connectivity Fingerprints. J. Chem. Inf. Model.
50, 742–754 (2010).
2.  Gobbi, A. & Poppinger, D. Genetic Optimization of Combinatorial Libraries.
Biotechnology and Bioengineering (Combinatorial Chemistry) 61, 47–54 (1998).
"[$([N;!H0;v3,v4&+1]),$([O,S;H1;+0]),n&H1&+0]", // Donor
"[$([O,S;H1;v2;!$(*-*=[O,N,P,S])]),$([O,S;H0;v2]),$([O,S;-]),
$([N;v3;!$(N-*=[O,N,P,S])]),n&H0&+0,
$([o,s;+0;!$([o,s]:n);!$([o,s]:c:n)])]", // Acceptor
"[a]", //Aromatic
"[F,Cl,Br,I]",//Halogen
"[#7;+,$([N;H2&+0][$([C,a]);!$([C,a](=O))]),
$([N;H1&+0]([$([C,a]);!$([C,a](=O))])[$([C,a]);!$([C,a](=O))]),
$([N;H0&+0]([C;!$(C(=O))])([C;!$(C(=O))])[C;!$(C(=O))])]", // Basic
"[$([C,S](=[O,S,P])-[O;H1,-1])]" //Acidic
Validation data
§  Diverse ChEMBL actives for 50 target classes1
§  Data taken from ChEMBL v14
§  Active : reported activity<10uM and confidence=9
§  Diverse: 100 actives picked using the RDKit’s implementation of the
MaxMin algorithm2 with radius 0 Morgan fingerprints (ECFP-like)
§  Inactives: 10000 molecules selected from the ZINC druglike set.
Selection criterion: two randomly selected neighbors (similarity via Morgan0
fingerprint>=0.5) for each of the 5000 actives
1.  Heikamp, K. & Bajorath, J. Large-Scale Similarity Search Profiling of ChEMBL
Compound Data Sets. JCIM 51, 1831–1839 (2011).
2.  Ashton, M. et al. Identification of Diverse Database Subsets using Property-Based
and Fragment-Based Molecular Descriptions. QSAR & Combinatorial Science 21,
598–604 (2002).
Validation procedure
§  Repeat 50 times for each data set:
•  Randomly pick 5 actives
•  Mix the remaining 95 actives with the 10K inactives
•  Rank that pool of compounds based on maximum similarity to the 5 actives
•  Calculate performance based on enrichment at 5% of the total dataset size
(10095)
§  Look at average enrichments within each assay
§  Compare the new fingerprint to other standard fingerprints;
MACCS, Morgan6 (bv + counts), Morgan4 (bv + counts), Morgan0 (bv +
counts), Topological Torsions (bv + counts), Atom Pairs (bv + counts), Avalon,
2D Pharmacophore, RDKit, 2 internal fingerprints
Results
The new fingerprint is the best for 29 of the 50 datasets
FeatureMorgan2
Morgan0
Back to the talk…
§  Reproducibility?
§  Requirements for reproducibility of published research
§  Practical aspects
Landrum, G. A. & Stiefl, N. Is that a scientific publication or an advertisement?
Reproducibility, source code and data in the computational chemistry literature. Future
Medicinal Chemistry 4, 1885–1887 (2012).
Reproducibility
Scientific publications have at least two goals: (i) to announce a result and (ii)
to convince readers that the result is correct. Mathematics papers are
expected to contain a proof complete enough to allow knowledgeable
readers to fill in any details. Papers in experimental science should describe
the results and provide a clear enough protocol to allow successful repetition
and extension.
Mesirov, J. P. Accessible Reproducible Research. Science 327,
415–416 (2010).
Reproducibility
An author’s central obligation is to present an accurate and complete account
of the research performed, absolutely avoiding deception, including the data
collected or used, as well as an objective discussion of the significance of the
research. Data are defined as information collected or used in generating
research conclusions. The research report and the data collected should
contain sufficient detail and reference to public sources of information to permit
a trained professional to reproduce the experimental observations.
ACS “Ethical Guidelines to Publication of Chemical Research”
Reproducibility
With these thoughts in mind, the editors of journals published by the American
Chemical Society now present a set of ethical guidelines for persons engaged
in the publication of chemical research, specifically, for editors, authors, and
manuscript reviewers. These guidelines are offered not in the sense that there
is any immediate crisis in ethical behavior, but rather from a conviction that the
observance of high ethical standards is so vital to the whole scientific enterprise
that a definition of those standards should be brought to the attention of all
concerned.
We believe that most of the guidelines now offered are already understood and
subscribed to by the majority of experienced research chemists. They may,
however, be of substantial help to those who are relatively new to research.
Even well-established scientists may appreciate an opportunity to review
matters so significant to the practice of science
ACS “Ethical Guidelines to Publication of Chemical Research”
Reproducibility
Experimental reproducibility is the coin of the scientific realm. The extent to
which measurements or observations agree when performed by different
individuals defines this important tenet of the scientific method. The formal
essence of experimental reproducibility was born of the philosophy of logical
positivism or logical empiricism, which purports to gain knowledge of the world
through the use of formal logic linked to observation. A key principle of logical
positivism is verificationism, which holds that every truth is verifiable by
experience. In this rational context, truth is defined by reproducible experience,
and unbiased scientific observation and determinism are its underpinnings.
…
The assumption that objectively true scientific observations must be reproducible
is implicit, yet direct tests of reproducibility are rarely found in the published
literature. This lack of published evidence of reproducibility stems from the
limited appeal of studies reproducing earlier work to most funding bodies and to
most editors. Furthermore, many readers of scientific journals— especially of
higher-impact journals—assume that if a study is of sufficient quality to pass the
scrutiny of rigorous reviewers, it must be true; this assumption is based on the
inferred equivalence of reproducibility and truth described above.
Loscalzo, J. Irreproducible Experimental Results: Causes,
(Mis)interpretations, and Consequences. Circulation 125, 1211–1214 (2012).
If it’s not reproducible science?
“Let me show you some cool pictures from my lab…”
Requirements for Reproducibility
§  Data used
§  Code/algorithm description
§  Results
Peng, R. D. Reproducible Research in Computational Science.
Science 334, 1226–1227 (2011).
Requirements for Reproducibility:
Data
§  This is a no brainer, right?
§  Unless it’s completely unprocessed (or the processing is part of the
detailed method description/code), it’s better to include the actual data
§  For sources like ChEMBL, a version number and SQL to grab the data
are probably adequate
§  “Ligands from PDB structures X, Y, and Z” probably not good enough
Requirements for Reproducibility:
Data
As a condition of publication, authors must agree to make available all data
necessary to understand and assess the conclusions of the manuscript to
any reader of Science. Data must be included in the body of the paper or in
the supplementary materials, where they can be viewed free of charge by all
visitors to the site. Certain types of data must be deposited in an approved
online database, including DNA and protein sequences, microarray data,
crystal structures, and climate records.
http://www.sciencemag.org/site/feature/contribinfo/faq/
index.xhtml#data_faq
Requirements for Reproducibility:
Data
§  What about chemical structures?
•  a table with drawings of molecules?
•  names instead of structures?
§  Why not include the structures in a machine-readable format?
This expanded use of electronic resources offers an excellent opportunity to make chemical
information more accessible and user-friendly to readers of scientific papers.
To take advantage of these opportunities, we have developed several online features that expand
the usefulness of chemical compound information for Nature Chemical Biology readers … In all
original research papers, compounds that are relevant to the background or results of the paper
are assigned a bolded, Arabic numeral that serves as a unique identifier for the compound. Each
numerical abbreviation in the HTML and PDF versions of the article is linked to a Compound Data
page, which shows the structure and the IUPAC or common name of the chemical compound.
From there, readers can download a ChemDraw file of the compound…To provide readers with
rapid access to all of the chemical compounds discussed in an article, we feature a Compound
Data Index page, which is accessible from the Compound Data page, the table of contents entry
for the paper, and the navigation tools on the right side of the Nature Chemical Biology website.
http://www.nature.com/nchembio/journal/v3/n6/full/nchembio0607-297.htm
Requirements for Reproducibility:
Chemical Data
From Nature Chemical Biology
Requirements for Reproducibility:
Code
Data and materials availability All data necessary to understand,
assess, and extend the conclusions of the manuscript must be available
to any reader of Science. All computer codes involved in the creation or
analysis of data must also be available to any reader of Science. After
publication, all reasonable requests for data and materials must be
fulfilled. Any restrictions on the availability of data, codes, or materials,
including fees and original data obtained from other sources (Materials
Transfer Agreements), must be disclosed to the editors upon submission.
http://www.sciencemag.org/site/feature/contribinfo/prep/
gen_info.xhtml#dataavail
Requirements for Reproducibility:
Code
An inherent principle of publication is that others should be able to
replicate and build upon the authors' published claims. Therefore, a
condition of publication in a Nature journal is that authors are required to
make materials, data and associated protocols promptly available to
readers without undue qualifications. Any restrictions on the availability of
materials or information must be disclosed to the editors at the time of
submission. Any restrictions must also be disclosed in the submitted
manuscript, including details of how readers can obtain materials and
information. If materials are to be distributed by a for-profit company, this
must be stated in the paper.
http://www.nature.com/authors/policies/availability.html
In the meantime, researchers must, when they are arranging the
commercialization of their work, bear in mind the implications that these
deals may have on their freedom to publish to the standards that the
community is entitled to expect.
http://www.nature.com/nature/journal/v442/
n7098/full/442001a.html
Requirements for Reproducibility:
Code
Ince, D. C., Hatton, L. & Graham-Cumming, J. The case for open
computer programs. Nature 482, 485–488 (2012).
We argue that, with some exceptions, anything less
than the release of source programs is intolerable for
results that depend on computation. The vagaries of
hardware, software and natural language will always
ensure that exact reproducibility remains uncertain, but
withholding code increases the chances that efforts to
reproduce results will fail.
Requirements for Reproducibility:
Code
§  “Black box” code sharing: installing the software on a publicly
accessible server, or providing executables for people to test
§  Does this help with reproducibility?
§  Not cut and dried. Needs discussion
Requirements for Reproducibility:
Results
§  Including the actual results is even more of a no brainer, right?
Homology Models of Human All-Trans Retinoic Acid Metabolizing Enzymes
CYP26B1 and CYP26B1 Spliced Variant
Homology models of CYP26B1 (cytochrome P450RAI2) and CYP26B1 spliced variant were
derived using the crystal structure of cyanobacterial CYP120A1 as template for the model building.
The quality of the homology models generated were carefully evaluated, and the natural substrate
all-trans-retinoic acid (atRA), several tetralone-derived retinoic acid metabolizing blocking agents
(RAMBAs), and a well-known potent inhibitor of CYP26B1 (R115866) were docked into the
homology model of full-length cytochrome P450 26B1. The results show that in the model of the
full-length CYP26B1, the protein is capable of distinguishing between the natural substrate (atRA),
R115866, and the tetralone derivatives. The spliced variant of CYP26B1 model displays a reduced
affinity for atRA compared to the full-length enzyme, in accordance with recently described
experimental information.
This paper, presenting two new homology models, does not
include either model.
Unfortunately I didn’t have to search long to find this example
How are we doing?
§  Survey of recent publications:
•  Everything in JCIM vol 52 #10
•  Everything in JCAMD vol 26 #10
•  Journal of Cheminformatics from July 2012-Nov 4 2012
§  Big differences between journals
§  Plenty of room for improvement
Journal	
   Type	
  of	
  paper	
   Count	
   Full	
  Data	
   Par3al	
  Data	
   Missing	
  Data	
   Code?	
  
JCIM	
   Method	
   13	
   6	
   3	
   4	
   1	
  
JCIM	
   Non-­‐method	
   16	
   10	
   3	
   3	
   0	
  
JCAMD	
   Method	
   3	
   3	
   0	
   0	
   0	
  
JCAMD	
   Non-­‐method	
   4	
   0	
   3	
   1	
   0	
  
JChemInf	
   Method	
   12	
   7	
   3	
   3	
   8	
  
JChemInf	
   Non-­‐method	
   3	
   0	
   0	
   0	
   0	
  
Practical considerations
§  Where to put the data and code?
•  Supplementary material
•  Code-sharing sites (sourceforge.net, google code, github)
•  Figshare
§  Considerations:
•  It needs to still be there 10+ years from now
•  Having a solid connection to the original paper is good
Tools for reproducible research
Knime
§  Open-source workflow tool
§  Strong data manipulation and mining capabilities
§  Data and results can be stored with the workflow.
Tools for reproducible research
IPython notebook
§  Python session running in a browser
•  Tab completion
•  Access to docstrings
§  Text formatting options available for including discussion or capturing
mathematics
§  Captures all data transformations and displays output
§  Tight integration with matplotlib
Tools for reproducible research
IPython notebook
Tools for reproducible research
IPython notebook
Tools for reproducible research
IPython notebook
Tools for reproducible research
IPython notebook
Back to the earlier interruption
§  Data? YES
§  Solid description of method? YES
§  Code? NO
Still ok, though, right?
Ooops
§  I had a typo in the script where I calculated EF_5 for the new
fingerprint:
§  Fixing that yields:
FeatureMorgan2
Morgan0
ef_5 = calcEnrichment(rankedSims,nActivesTotal=80)
The new fingerprint is no
better than the others.
Should be
95
Requirements for Reproducibility
§  Data used
§  Code/algorithm description
§  Results
Perhaps the biggest barrier to reproducible
research is the lack of a deeply ingrained
culture that simply requires reproducibility for
all scientific claims.
Peng, R. D. Reproducible Research in Computational Science.
Science 334, 1226–1227 (2011).
Acknowledgements
§  NIBR:
•  Nik Stiefl (GDC/CADD)
•  Nikolas Fechner (NIBR IT/IS Sigma)
•  Sereina Riniker (NIBR IT/IS Sigma)
§  Matthias Rarey

More Related Content

What's hot (20)

PDF
Is one enough? Data warehousing for biomedical research
Greg Landrum
 
PPT
Gene Ontology Enrichment Network Analysis -Tutorial
Dmitry Grapov
 
PDF
Drug Discovery and Development Using AI
Databricks
 
PPTX
Knowledge graph applications for cosmetics industry
Anton Yuryev
 
PDF
Mining 'Bigger' Datasets to Create, Validate and Share Machine Learning Models
Sean Ekins
 
PDF
Drug Repurposing using Deep Learning on Knowledge Graphs
Databricks
 
PDF
AXP302
Abhishek Pai
 
PDF
Reaxys rmc unified platform_ webinar_
Ann-Marie Roche
 
PDF
Deep Learning on nVidia GPUs for QSAR, QSPR and QNAR predictions
Valery Tkachenko
 
PPTX
An examination of data quality on QSAR Modeling in regards to the environment...
US Environmental Protection Agency (EPA), Center for Computational Toxicology and Exposure
 
PPTX
Medicinal Chemistry Due Diligence: Computational Predictions of an expert’s e...
Sean Ekins
 
PPT
Aiding Computer Aided Drug Design
Shahir Shamsir
 
PDF
Why are we still doing industrial age drug
Sean Ekins
 
PDF
Reproducible research: First steps.
Richard Layton
 
PPTX
2015 balti-and-bioinformatics
c.titus.brown
 
PPTX
Resolving cryptic needles to molecular structures: The GtoPdb experience
Chris Southan
 
PPTX
Multi-omics methods and resources for Bioconductor
Levi Waldron
 
PDF
II-SDV 2015, 20 - 21 April, in Nice
Dr. Haxel Consult
 
PDF
CINF 29: Visualization and manipulation of Matched Molecular Series for decis...
NextMove Software
 
PDF
OpenTox Europe 2013
Alejandra Gonzalez-Beltran
 
Is one enough? Data warehousing for biomedical research
Greg Landrum
 
Gene Ontology Enrichment Network Analysis -Tutorial
Dmitry Grapov
 
Drug Discovery and Development Using AI
Databricks
 
Knowledge graph applications for cosmetics industry
Anton Yuryev
 
Mining 'Bigger' Datasets to Create, Validate and Share Machine Learning Models
Sean Ekins
 
Drug Repurposing using Deep Learning on Knowledge Graphs
Databricks
 
AXP302
Abhishek Pai
 
Reaxys rmc unified platform_ webinar_
Ann-Marie Roche
 
Deep Learning on nVidia GPUs for QSAR, QSPR and QNAR predictions
Valery Tkachenko
 
An examination of data quality on QSAR Modeling in regards to the environment...
US Environmental Protection Agency (EPA), Center for Computational Toxicology and Exposure
 
Medicinal Chemistry Due Diligence: Computational Predictions of an expert’s e...
Sean Ekins
 
Aiding Computer Aided Drug Design
Shahir Shamsir
 
Why are we still doing industrial age drug
Sean Ekins
 
Reproducible research: First steps.
Richard Layton
 
2015 balti-and-bioinformatics
c.titus.brown
 
Resolving cryptic needles to molecular structures: The GtoPdb experience
Chris Southan
 
Multi-omics methods and resources for Bioconductor
Levi Waldron
 
II-SDV 2015, 20 - 21 April, in Nice
Dr. Haxel Consult
 
CINF 29: Visualization and manipulation of Matched Molecular Series for decis...
NextMove Software
 
OpenTox Europe 2013
Alejandra Gonzalez-Beltran
 

Viewers also liked (20)

PPT
CSS Media Queries (WordCamp 2010)
Michael Jendryschik
 
PDF
Machine learning in the life sciences with knime
Greg Landrum
 
PDF
Open-source from/in the enterprise: the RDKit
Greg Landrum
 
PDF
Getting started
Hitesh Sharma
 
PDF
How To Treat A Bleeding Wound From A Popped Pimple
M. David Cole, MD
 
PDF
Scottish Public Opinion Monitor: Gordon's Fightback
Ipsos UK
 
PPTX
Presentación sobre desarrollo de nuevos negocios con Grupo Supernova y Cedice...
Alejandro Bermudez
 
PDF
What About Semantics? - Stefan Gradmann, WWW2012, Lyon, France
Digitised Manuscripts to Europeana
 
PPTX
Congreso eucarístico vale neme y ferchu albañil
ferchualba
 
PDF
Inclusión laboral. empleo y discapacidad.
José María
 
PDF
Las políticas sobre discapacidad en el sistema universitario español.
José María
 
PDF
TREAT YOUR COMPUTER NECK
Eason Chan
 
PPTX
Test ppt
Hitesh Sharma
 
PDF
Atividades de Natal adaptadas em spc
Madalena Charruadas
 
PPT
Introducción a la computadora - Parte I
Manuel Otero
 
PPTX
Johan Stuve, presentación de servicios
Johan Stuve
 
PPT
Upliftment of man based on character
jasvirsandhu
 
PDF
Folha de setembro
TecnologiaPMC
 
PDF
3 d pie chart circular with hole in center 10 stages powerpoint presentation ...
SlideTeam.net
 
PPT
Introduction To Ant1
Rajesh Kumar
 
CSS Media Queries (WordCamp 2010)
Michael Jendryschik
 
Machine learning in the life sciences with knime
Greg Landrum
 
Open-source from/in the enterprise: the RDKit
Greg Landrum
 
Getting started
Hitesh Sharma
 
How To Treat A Bleeding Wound From A Popped Pimple
M. David Cole, MD
 
Scottish Public Opinion Monitor: Gordon's Fightback
Ipsos UK
 
Presentación sobre desarrollo de nuevos negocios con Grupo Supernova y Cedice...
Alejandro Bermudez
 
What About Semantics? - Stefan Gradmann, WWW2012, Lyon, France
Digitised Manuscripts to Europeana
 
Congreso eucarístico vale neme y ferchu albañil
ferchualba
 
Inclusión laboral. empleo y discapacidad.
José María
 
Las políticas sobre discapacidad en el sistema universitario español.
José María
 
TREAT YOUR COMPUTER NECK
Eason Chan
 
Test ppt
Hitesh Sharma
 
Atividades de Natal adaptadas em spc
Madalena Charruadas
 
Introducción a la computadora - Parte I
Manuel Otero
 
Johan Stuve, presentación de servicios
Johan Stuve
 
Upliftment of man based on character
jasvirsandhu
 
Folha de setembro
TecnologiaPMC
 
3 d pie chart circular with hole in center 10 stages powerpoint presentation ...
SlideTeam.net
 
Introduction To Ant1
Rajesh Kumar
 
Ad

Similar to Reproducibility in cheminformatics and computational chemistry research: certainly we can do better than this (20)

PDF
Is that a scientific report or just some cool pictures from the lab? Reproduc...
Greg Landrum
 
PDF
Reproducibility by Other Means: Transparent Research Objects
Timothy McPhillips
 
PPTX
Reproducible research: theory
C. Tobin Magle
 
PPT
Results may vary: Collaborations Workshop, Oxford 2014
Carole Goble
 
PPTX
Research Objects for FAIRer Science
Carole Goble
 
PDF
Reproducible research in molecular biophysics and structural biology
khinsen
 
PPTX
Being Reproducible: SSBSS Summer School 2017
Carole Goble
 
PDF
Five selfish reasons to work reproducibly
Florian Markowetz
 
PPT
Repeatability and Reproducibility in science
pramod41kumar
 
PPTX
Reproducibility and Scientific Research: why, what, where, when, who, how
Carole Goble
 
PDF
Open & reproducible research - What can we do in practice?
Felix Z. Hoffmann
 
PPT
ISMB/ECCB 2013 Keynote Goble Results may vary: what is reproducible? why do o...
Carole Goble
 
PPTX
The habits of highly successful data:
Anita de Waard
 
PPTX
What is Reproducibility? The R* brouhaha (and how Research Objects can help)
Carole Goble
 
PPTX
What is Reproducibility? The R* brouhaha and how Research Objects can help
Carole Goble
 
PPT
Our dire need to mandate data standards and expectations for scientific publi...
US Environmental Protection Agency (EPA), Center for Computational Toxicology and Exposure
 
PPTX
Marco Roos: Newton's ideas and methods are preserved forever: how about yours?
GigaScience, BGI Hong Kong
 
PDF
Reproducibility in computer-assisted research
khinsen
 
PPTX
Towards Reproducible Science: a few building blocks from my personal experience
Oscar Corcho
 
PDF
Is the current measure of excellence perverting Science? A Data deluge is com...
Lourdes Verdes-Montenegro
 
Is that a scientific report or just some cool pictures from the lab? Reproduc...
Greg Landrum
 
Reproducibility by Other Means: Transparent Research Objects
Timothy McPhillips
 
Reproducible research: theory
C. Tobin Magle
 
Results may vary: Collaborations Workshop, Oxford 2014
Carole Goble
 
Research Objects for FAIRer Science
Carole Goble
 
Reproducible research in molecular biophysics and structural biology
khinsen
 
Being Reproducible: SSBSS Summer School 2017
Carole Goble
 
Five selfish reasons to work reproducibly
Florian Markowetz
 
Repeatability and Reproducibility in science
pramod41kumar
 
Reproducibility and Scientific Research: why, what, where, when, who, how
Carole Goble
 
Open & reproducible research - What can we do in practice?
Felix Z. Hoffmann
 
ISMB/ECCB 2013 Keynote Goble Results may vary: what is reproducible? why do o...
Carole Goble
 
The habits of highly successful data:
Anita de Waard
 
What is Reproducibility? The R* brouhaha (and how Research Objects can help)
Carole Goble
 
What is Reproducibility? The R* brouhaha and how Research Objects can help
Carole Goble
 
Our dire need to mandate data standards and expectations for scientific publi...
US Environmental Protection Agency (EPA), Center for Computational Toxicology and Exposure
 
Marco Roos: Newton's ideas and methods are preserved forever: how about yours?
GigaScience, BGI Hong Kong
 
Reproducibility in computer-assisted research
khinsen
 
Towards Reproducible Science: a few building blocks from my personal experience
Oscar Corcho
 
Is the current measure of excellence perverting Science? A Data deluge is com...
Lourdes Verdes-Montenegro
 
Ad

More from Greg Landrum (13)

PDF
Chemical registration
Greg Landrum
 
PDF
Mike Lynch Award Lecture, ICCS 2022
Greg Landrum
 
PDF
Google BigQuery for analysis of scientific datasets: Interactive exploration ...
Greg Landrum
 
PDF
ACS San Diego - The RDKit: Open-source cheminformatics
Greg Landrum
 
PDF
Building useful models for imbalanced datasets (without resampling)
Greg Landrum
 
PDF
Moving from Artisanal to Industrial Machine Learning
Greg Landrum
 
PDF
Building useful models for imbalanced datasets (without resampling)
Greg Landrum
 
PDF
Let’s talk about reproducible data analysis
Greg Landrum
 
PDF
How Do You Build and Validate 1500 Models and What Can You Learn from Them?
Greg Landrum
 
PDF
Interactive and reproducible data analysis with the open-source KNIME Analyti...
Greg Landrum
 
PDF
Processing malaria HTS results using KNIME: a tutorial
Greg Landrum
 
PDF
Big (chemical) data? No Problem!
Greg Landrum
 
PDF
Some "challenges" on the open-source/open-data front
Greg Landrum
 
Chemical registration
Greg Landrum
 
Mike Lynch Award Lecture, ICCS 2022
Greg Landrum
 
Google BigQuery for analysis of scientific datasets: Interactive exploration ...
Greg Landrum
 
ACS San Diego - The RDKit: Open-source cheminformatics
Greg Landrum
 
Building useful models for imbalanced datasets (without resampling)
Greg Landrum
 
Moving from Artisanal to Industrial Machine Learning
Greg Landrum
 
Building useful models for imbalanced datasets (without resampling)
Greg Landrum
 
Let’s talk about reproducible data analysis
Greg Landrum
 
How Do You Build and Validate 1500 Models and What Can You Learn from Them?
Greg Landrum
 
Interactive and reproducible data analysis with the open-source KNIME Analyti...
Greg Landrum
 
Processing malaria HTS results using KNIME: a tutorial
Greg Landrum
 
Big (chemical) data? No Problem!
Greg Landrum
 
Some "challenges" on the open-source/open-data front
Greg Landrum
 

Recently uploaded (20)

PPTX
Applied-Statistics-Mastering-Data-Driven-Decisions.pptx
parmaryashparmaryash
 
PDF
Responsible AI and AI Ethics - By Sylvester Ebhonu
Sylvester Ebhonu
 
PPTX
AI Code Generation Risks (Ramkumar Dilli, CIO, Myridius)
Priyanka Aash
 
PPTX
Simple and concise overview about Quantum computing..pptx
mughal641
 
PDF
RAT Builders - How to Catch Them All [DeepSec 2024]
malmoeb
 
PPTX
AVL ( audio, visuals or led ), technology.
Rajeshwri Panchal
 
PDF
Presentation about Hardware and Software in Computer
snehamodhawadiya
 
PDF
Peak of Data & AI Encore - Real-Time Insights & Scalable Editing with ArcGIS
Safe Software
 
PDF
The Future of Mobile Is Context-Aware—Are You Ready?
iProgrammer Solutions Private Limited
 
PDF
Build with AI and GDG Cloud Bydgoszcz- ADK .pdf
jaroslawgajewski1
 
PDF
OFFOFFBOX™ – A New Era for African Film | Startup Presentation
ambaicciwalkerbrian
 
PPTX
Agentic AI in Healthcare Driving the Next Wave of Digital Transformation
danielle hunter
 
PPTX
Dev Dives: Automate, test, and deploy in one place—with Unified Developer Exp...
AndreeaTom
 
PDF
Google I/O Extended 2025 Baku - all ppts
HusseinMalikMammadli
 
PPTX
Agile Chennai 18-19 July 2025 | Emerging patterns in Agentic AI by Bharani Su...
AgileNetwork
 
PDF
Market Insight : ETH Dominance Returns
CIFDAQ
 
PDF
GDG Cloud Munich - Intro - Luiz Carneiro - #BuildWithAI - July - Abdel.pdf
Luiz Carneiro
 
PPTX
AI and Robotics for Human Well-being.pptx
JAYMIN SUTHAR
 
PPTX
OA presentation.pptx OA presentation.pptx
pateldhruv002338
 
PDF
Research-Fundamentals-and-Topic-Development.pdf
ayesha butalia
 
Applied-Statistics-Mastering-Data-Driven-Decisions.pptx
parmaryashparmaryash
 
Responsible AI and AI Ethics - By Sylvester Ebhonu
Sylvester Ebhonu
 
AI Code Generation Risks (Ramkumar Dilli, CIO, Myridius)
Priyanka Aash
 
Simple and concise overview about Quantum computing..pptx
mughal641
 
RAT Builders - How to Catch Them All [DeepSec 2024]
malmoeb
 
AVL ( audio, visuals or led ), technology.
Rajeshwri Panchal
 
Presentation about Hardware and Software in Computer
snehamodhawadiya
 
Peak of Data & AI Encore - Real-Time Insights & Scalable Editing with ArcGIS
Safe Software
 
The Future of Mobile Is Context-Aware—Are You Ready?
iProgrammer Solutions Private Limited
 
Build with AI and GDG Cloud Bydgoszcz- ADK .pdf
jaroslawgajewski1
 
OFFOFFBOX™ – A New Era for African Film | Startup Presentation
ambaicciwalkerbrian
 
Agentic AI in Healthcare Driving the Next Wave of Digital Transformation
danielle hunter
 
Dev Dives: Automate, test, and deploy in one place—with Unified Developer Exp...
AndreeaTom
 
Google I/O Extended 2025 Baku - all ppts
HusseinMalikMammadli
 
Agile Chennai 18-19 July 2025 | Emerging patterns in Agentic AI by Bharani Su...
AgileNetwork
 
Market Insight : ETH Dominance Returns
CIFDAQ
 
GDG Cloud Munich - Intro - Luiz Carneiro - #BuildWithAI - July - Abdel.pdf
Luiz Carneiro
 
AI and Robotics for Human Well-being.pptx
JAYMIN SUTHAR
 
OA presentation.pptx OA presentation.pptx
pateldhruv002338
 
Research-Fundamentals-and-Topic-Development.pdf
ayesha butalia
 

Reproducibility in cheminformatics and computational chemistry research: certainly we can do better than this

  • 1. Reproducibility in cheminformatics and computational chemistry research: Certainly we can do better than this Gregory Landrum Ph.D. NIBR IT Novartis Institutes for BioMedical Research Basel GCC 2012 Goslar
  • 2. Outline §  Reproducibility? §  Requirements for reproducibility of published research §  Practical aspects Landrum, G. A. & Stiefl, N. Is that a scientific publication or an advertisement? Reproducibility, source code and data in the computational chemistry literature. Future Medicinal Chemistry 4, 1885–1887 (2012).
  • 4. A new fingerprint for similarity-based virtual screening §  Start with Morgan fingerprints (a.k.a. circular fingerprints1) §  The usual FCFP algorithm uses fairly crude feature definitions §  Combine the RDKit Morgan fingerprint algorithm with pharmacophoric features calculated using “better” feature definitions2. 1.  Rogers, D. & Hahn, M. Extended-Connectivity Fingerprints. J. Chem. Inf. Model. 50, 742–754 (2010). 2.  Gobbi, A. & Poppinger, D. Genetic Optimization of Combinatorial Libraries. Biotechnology and Bioengineering (Combinatorial Chemistry) 61, 47–54 (1998). "[$([N;!H0;v3,v4&+1]),$([O,S;H1;+0]),n&H1&+0]", // Donor "[$([O,S;H1;v2;!$(*-*=[O,N,P,S])]),$([O,S;H0;v2]),$([O,S;-]), $([N;v3;!$(N-*=[O,N,P,S])]),n&H0&+0, $([o,s;+0;!$([o,s]:n);!$([o,s]:c:n)])]", // Acceptor "[a]", //Aromatic "[F,Cl,Br,I]",//Halogen "[#7;+,$([N;H2&+0][$([C,a]);!$([C,a](=O))]), $([N;H1&+0]([$([C,a]);!$([C,a](=O))])[$([C,a]);!$([C,a](=O))]), $([N;H0&+0]([C;!$(C(=O))])([C;!$(C(=O))])[C;!$(C(=O))])]", // Basic "[$([C,S](=[O,S,P])-[O;H1,-1])]" //Acidic
  • 5. Validation data §  Diverse ChEMBL actives for 50 target classes1 §  Data taken from ChEMBL v14 §  Active : reported activity<10uM and confidence=9 §  Diverse: 100 actives picked using the RDKit’s implementation of the MaxMin algorithm2 with radius 0 Morgan fingerprints (ECFP-like) §  Inactives: 10000 molecules selected from the ZINC druglike set. Selection criterion: two randomly selected neighbors (similarity via Morgan0 fingerprint>=0.5) for each of the 5000 actives 1.  Heikamp, K. & Bajorath, J. Large-Scale Similarity Search Profiling of ChEMBL Compound Data Sets. JCIM 51, 1831–1839 (2011). 2.  Ashton, M. et al. Identification of Diverse Database Subsets using Property-Based and Fragment-Based Molecular Descriptions. QSAR & Combinatorial Science 21, 598–604 (2002).
  • 6. Validation procedure §  Repeat 50 times for each data set: •  Randomly pick 5 actives •  Mix the remaining 95 actives with the 10K inactives •  Rank that pool of compounds based on maximum similarity to the 5 actives •  Calculate performance based on enrichment at 5% of the total dataset size (10095) §  Look at average enrichments within each assay §  Compare the new fingerprint to other standard fingerprints; MACCS, Morgan6 (bv + counts), Morgan4 (bv + counts), Morgan0 (bv + counts), Topological Torsions (bv + counts), Atom Pairs (bv + counts), Avalon, 2D Pharmacophore, RDKit, 2 internal fingerprints
  • 7. Results The new fingerprint is the best for 29 of the 50 datasets FeatureMorgan2 Morgan0
  • 8. Back to the talk… §  Reproducibility? §  Requirements for reproducibility of published research §  Practical aspects Landrum, G. A. & Stiefl, N. Is that a scientific publication or an advertisement? Reproducibility, source code and data in the computational chemistry literature. Future Medicinal Chemistry 4, 1885–1887 (2012).
  • 9. Reproducibility Scientific publications have at least two goals: (i) to announce a result and (ii) to convince readers that the result is correct. Mathematics papers are expected to contain a proof complete enough to allow knowledgeable readers to fill in any details. Papers in experimental science should describe the results and provide a clear enough protocol to allow successful repetition and extension. Mesirov, J. P. Accessible Reproducible Research. Science 327, 415–416 (2010).
  • 10. Reproducibility An author’s central obligation is to present an accurate and complete account of the research performed, absolutely avoiding deception, including the data collected or used, as well as an objective discussion of the significance of the research. Data are defined as information collected or used in generating research conclusions. The research report and the data collected should contain sufficient detail and reference to public sources of information to permit a trained professional to reproduce the experimental observations. ACS “Ethical Guidelines to Publication of Chemical Research”
  • 11. Reproducibility With these thoughts in mind, the editors of journals published by the American Chemical Society now present a set of ethical guidelines for persons engaged in the publication of chemical research, specifically, for editors, authors, and manuscript reviewers. These guidelines are offered not in the sense that there is any immediate crisis in ethical behavior, but rather from a conviction that the observance of high ethical standards is so vital to the whole scientific enterprise that a definition of those standards should be brought to the attention of all concerned. We believe that most of the guidelines now offered are already understood and subscribed to by the majority of experienced research chemists. They may, however, be of substantial help to those who are relatively new to research. Even well-established scientists may appreciate an opportunity to review matters so significant to the practice of science ACS “Ethical Guidelines to Publication of Chemical Research”
  • 12. Reproducibility Experimental reproducibility is the coin of the scientific realm. The extent to which measurements or observations agree when performed by different individuals defines this important tenet of the scientific method. The formal essence of experimental reproducibility was born of the philosophy of logical positivism or logical empiricism, which purports to gain knowledge of the world through the use of formal logic linked to observation. A key principle of logical positivism is verificationism, which holds that every truth is verifiable by experience. In this rational context, truth is defined by reproducible experience, and unbiased scientific observation and determinism are its underpinnings. … The assumption that objectively true scientific observations must be reproducible is implicit, yet direct tests of reproducibility are rarely found in the published literature. This lack of published evidence of reproducibility stems from the limited appeal of studies reproducing earlier work to most funding bodies and to most editors. Furthermore, many readers of scientific journals— especially of higher-impact journals—assume that if a study is of sufficient quality to pass the scrutiny of rigorous reviewers, it must be true; this assumption is based on the inferred equivalence of reproducibility and truth described above. Loscalzo, J. Irreproducible Experimental Results: Causes, (Mis)interpretations, and Consequences. Circulation 125, 1211–1214 (2012).
  • 13. If it’s not reproducible science? “Let me show you some cool pictures from my lab…”
  • 14. Requirements for Reproducibility §  Data used §  Code/algorithm description §  Results Peng, R. D. Reproducible Research in Computational Science. Science 334, 1226–1227 (2011).
  • 15. Requirements for Reproducibility: Data §  This is a no brainer, right? §  Unless it’s completely unprocessed (or the processing is part of the detailed method description/code), it’s better to include the actual data §  For sources like ChEMBL, a version number and SQL to grab the data are probably adequate §  “Ligands from PDB structures X, Y, and Z” probably not good enough
  • 16. Requirements for Reproducibility: Data As a condition of publication, authors must agree to make available all data necessary to understand and assess the conclusions of the manuscript to any reader of Science. Data must be included in the body of the paper or in the supplementary materials, where they can be viewed free of charge by all visitors to the site. Certain types of data must be deposited in an approved online database, including DNA and protein sequences, microarray data, crystal structures, and climate records. http://www.sciencemag.org/site/feature/contribinfo/faq/ index.xhtml#data_faq
  • 17. Requirements for Reproducibility: Data §  What about chemical structures? •  a table with drawings of molecules? •  names instead of structures? §  Why not include the structures in a machine-readable format? This expanded use of electronic resources offers an excellent opportunity to make chemical information more accessible and user-friendly to readers of scientific papers. To take advantage of these opportunities, we have developed several online features that expand the usefulness of chemical compound information for Nature Chemical Biology readers … In all original research papers, compounds that are relevant to the background or results of the paper are assigned a bolded, Arabic numeral that serves as a unique identifier for the compound. Each numerical abbreviation in the HTML and PDF versions of the article is linked to a Compound Data page, which shows the structure and the IUPAC or common name of the chemical compound. From there, readers can download a ChemDraw file of the compound…To provide readers with rapid access to all of the chemical compounds discussed in an article, we feature a Compound Data Index page, which is accessible from the Compound Data page, the table of contents entry for the paper, and the navigation tools on the right side of the Nature Chemical Biology website. http://www.nature.com/nchembio/journal/v3/n6/full/nchembio0607-297.htm
  • 18. Requirements for Reproducibility: Chemical Data From Nature Chemical Biology
  • 19. Requirements for Reproducibility: Code Data and materials availability All data necessary to understand, assess, and extend the conclusions of the manuscript must be available to any reader of Science. All computer codes involved in the creation or analysis of data must also be available to any reader of Science. After publication, all reasonable requests for data and materials must be fulfilled. Any restrictions on the availability of data, codes, or materials, including fees and original data obtained from other sources (Materials Transfer Agreements), must be disclosed to the editors upon submission. http://www.sciencemag.org/site/feature/contribinfo/prep/ gen_info.xhtml#dataavail
  • 20. Requirements for Reproducibility: Code An inherent principle of publication is that others should be able to replicate and build upon the authors' published claims. Therefore, a condition of publication in a Nature journal is that authors are required to make materials, data and associated protocols promptly available to readers without undue qualifications. Any restrictions on the availability of materials or information must be disclosed to the editors at the time of submission. Any restrictions must also be disclosed in the submitted manuscript, including details of how readers can obtain materials and information. If materials are to be distributed by a for-profit company, this must be stated in the paper. http://www.nature.com/authors/policies/availability.html In the meantime, researchers must, when they are arranging the commercialization of their work, bear in mind the implications that these deals may have on their freedom to publish to the standards that the community is entitled to expect. http://www.nature.com/nature/journal/v442/ n7098/full/442001a.html
  • 21. Requirements for Reproducibility: Code Ince, D. C., Hatton, L. & Graham-Cumming, J. The case for open computer programs. Nature 482, 485–488 (2012). We argue that, with some exceptions, anything less than the release of source programs is intolerable for results that depend on computation. The vagaries of hardware, software and natural language will always ensure that exact reproducibility remains uncertain, but withholding code increases the chances that efforts to reproduce results will fail.
  • 22. Requirements for Reproducibility: Code §  “Black box” code sharing: installing the software on a publicly accessible server, or providing executables for people to test §  Does this help with reproducibility? §  Not cut and dried. Needs discussion
  • 23. Requirements for Reproducibility: Results §  Including the actual results is even more of a no brainer, right? Homology Models of Human All-Trans Retinoic Acid Metabolizing Enzymes CYP26B1 and CYP26B1 Spliced Variant Homology models of CYP26B1 (cytochrome P450RAI2) and CYP26B1 spliced variant were derived using the crystal structure of cyanobacterial CYP120A1 as template for the model building. The quality of the homology models generated were carefully evaluated, and the natural substrate all-trans-retinoic acid (atRA), several tetralone-derived retinoic acid metabolizing blocking agents (RAMBAs), and a well-known potent inhibitor of CYP26B1 (R115866) were docked into the homology model of full-length cytochrome P450 26B1. The results show that in the model of the full-length CYP26B1, the protein is capable of distinguishing between the natural substrate (atRA), R115866, and the tetralone derivatives. The spliced variant of CYP26B1 model displays a reduced affinity for atRA compared to the full-length enzyme, in accordance with recently described experimental information. This paper, presenting two new homology models, does not include either model. Unfortunately I didn’t have to search long to find this example
  • 24. How are we doing? §  Survey of recent publications: •  Everything in JCIM vol 52 #10 •  Everything in JCAMD vol 26 #10 •  Journal of Cheminformatics from July 2012-Nov 4 2012 §  Big differences between journals §  Plenty of room for improvement Journal   Type  of  paper   Count   Full  Data   Par3al  Data   Missing  Data   Code?   JCIM   Method   13   6   3   4   1   JCIM   Non-­‐method   16   10   3   3   0   JCAMD   Method   3   3   0   0   0   JCAMD   Non-­‐method   4   0   3   1   0   JChemInf   Method   12   7   3   3   8   JChemInf   Non-­‐method   3   0   0   0   0  
  • 25. Practical considerations §  Where to put the data and code? •  Supplementary material •  Code-sharing sites (sourceforge.net, google code, github) •  Figshare §  Considerations: •  It needs to still be there 10+ years from now •  Having a solid connection to the original paper is good
  • 26. Tools for reproducible research Knime §  Open-source workflow tool §  Strong data manipulation and mining capabilities §  Data and results can be stored with the workflow.
  • 27. Tools for reproducible research IPython notebook §  Python session running in a browser •  Tab completion •  Access to docstrings §  Text formatting options available for including discussion or capturing mathematics §  Captures all data transformations and displays output §  Tight integration with matplotlib
  • 28. Tools for reproducible research IPython notebook
  • 29. Tools for reproducible research IPython notebook
  • 30. Tools for reproducible research IPython notebook
  • 31. Tools for reproducible research IPython notebook
  • 32. Back to the earlier interruption §  Data? YES §  Solid description of method? YES §  Code? NO Still ok, though, right?
  • 33. Ooops §  I had a typo in the script where I calculated EF_5 for the new fingerprint: §  Fixing that yields: FeatureMorgan2 Morgan0 ef_5 = calcEnrichment(rankedSims,nActivesTotal=80) The new fingerprint is no better than the others. Should be 95
  • 34. Requirements for Reproducibility §  Data used §  Code/algorithm description §  Results
  • 35. Perhaps the biggest barrier to reproducible research is the lack of a deeply ingrained culture that simply requires reproducibility for all scientific claims. Peng, R. D. Reproducible Research in Computational Science. Science 334, 1226–1227 (2011).
  • 36. Acknowledgements §  NIBR: •  Nik Stiefl (GDC/CADD) •  Nikolas Fechner (NIBR IT/IS Sigma) •  Sereina Riniker (NIBR IT/IS Sigma) §  Matthias Rarey