Knime & bioinformatics

Copyright © 2015 KNIME.com AG
Биоинформатик в тридевятом
царстве, или двое программистов из
ларца KNIME
Oleg Yasnev
KNIME.com

Copyright © 2015 KNIME.com AG 2
А вы что же за
меня и код
писать будете?
Ага!
Кадр из мультфильма «Вовка в тридевятом царстве» © «Союзмультфильм»

KNIME.com
3

KNIME.com
• KNIME.com founded in 2008
• Offices in Zurich, San Francisco (Aug ‘13), Berlin
(May ‘14) and Konstanz (October ‘15)
• 15 open source releases, 10 product releases (in
2014)
• >2m lines of code
• 600k lines of community code
4

Advanced
Analytics
Pharma
Health Care
Finance
Retail
Customer
Intelligence
Manu-
facturing
Broad Range of KNIME Application Areas
5

The KNIME Analytics Platform
6

From Access to Visualization and Deployment

Data Access
• Databases
– MySQL, PostgreSQL
– any JDBC (Oracle, DB2, MS SQL
Server)
• Files
– Csv, txt
– Excel, Word, PDF
– SAS, SPSS
– XML
– PMML
– Images, texts, networks, chem
• Web, Cloud
– REST, Web services
– Twitter, Google

Big Data
• HDFS support
• Hive
• Impala
• HP Vertica
• In-database processing

Transformation
• Preprocessing
– Row, column, matrix based
• Data blending
– Join, concatenate, append
• Aggregation
– Grouping, pivoting, binning
• Feature Creation and
Selection

Analyze & Data Mining
• Regression
– Linear, Logistic
• Classification
– Decision tree, ensembles,
SVM, MLP, Naïve Bayes
• Clustering
– k-means, DBSCAN, hierarchical
• Validation
– Cross-validation, scoring, ROC
• Misc
– PCA, MDS, item set mining
• External
– R, Weka

Visualization
• Interactive
– Scatter plot, histogram, pie
charts, box plot
– Highlighting (brushing)
• JFreeChart
• JavaScript
• Misc
– Tag cloud, open street map,
networks, molecules
• External
– R

Deployment
• Database
• Files
– Excel, csv, txt
– XML
– PMML
– to: local, KNIME Server,
SSH-, FTP-Server
• BIRT Reporting

Statistics
Data Mining
Machine Learning
Web Analytics
Text Mining
Network Analysis
Social Media Analysis
WEKA
R
Community / 3rd
MySQL, Oracle, etc.
SAS, SPSS, etc.
Excel, Flat, etc.
Hive etc.
XML, PMML
Text, Doc, Image
Web Crawlers
Industry Specific
Community / 3rd
ETL
Row,
Column
Matrix
Text, Image
Time Series
Java
Python
Community / 3rd
R
JFreeChart
Community / 3rd
via BIRT
PMML
XML
Databases
Excel, Flat, etc.
Hive etc.
Text, Doc, Image
Industry Specific
Community / 3rd
Over 1000 native and embedded nodes included:
14

KNIME: Integrating Data and Tools
15

Big Data.
Pre-processing on Hadoop

In-Database Processing
17
Loads your pre-processed
data into KNIME

Reader/Writer
• Table selection
• Load data into KNIME
• Create table as select
• Insert/append data
• Delete rows from table
• Update values in table
18

Hive/Impala Loader
• Upload a KNIME data table to Hive/Impala
• Part of the commercial Big Data Extension
19

Manipulation
• Filter rows and
columns
• Join tables/queries
• Sort your data
• Write your own query
• Aggregate your data
20

Database GroupBy – Manual Aggregation
21

Database GroupBy – Type Based Aggregation
22
Matches
all cells
Matches
all numeric
cells

Utility
• Drop table
– missing table handling
– cascade option
• Execute any SQL
statement e.g. DDL
• Manipulate existing
queries
23

HDFS File Handling
• New nodes
– HDFS Connection
– HDFS File Permission
• Utilize the existing remote
file handling nodes
– Upload/download files
– Create/list directories
– Delete files
24

HDFS File Handling
25

Workflow 1: PrepareData
26
~ 2 days
Irish Smart Energy Meter Trials
• July 2009 – Dec 2010
• 6000 meters
• roughly 176m rows of data

Import Data from Database into KNIME
27
< 30 min

Big Data.
Machine Learning on Hadoop

Machine Learning on Hadoop
• Based on Spark MLlib
• Scalable machine learning library
• Runs on Hadoop
• Algorithms for
– Classification (decision tree, naïve bayes, …)
– Regression (logistic regression, linear regression, …)
– Clustering (k-means)
– Collaborative filtering (ALS)
– Dimensionality reduction (SVD, PCA)
29

MLlib Integration
• Usage model and dialogs similar to existing nodes
• No coding required

MLlib Integration
• MLlib model ports for model transfer
• Native MLlib model learning and prediction
• Spark nodes start and manage Spark jobs
• Supports Spark job cancelation
Native MLlib model

MLlib Integration
• Spark RDDs as input/output format
• Data stays within your cluster
• No unnecessary data movements
• Several input/output nodes e.g. Hive, hdfs files, …

Mass Learning – Fast Event Prediction
• Convert supported MLlib models to PMML
• Mass learning on Hadoop
• Fast event prediction based on compiled models

Mix and Match
• Combine with existing KNIME nodes

Modularize and Execute Your Own Spark Code

Spark Node Overview

А что же Rocket Science?
38

Community Contributors
39
Technology
Partners
Distribution
& Consulting
Partners
Community
Contributors
Community
User Base
Donated by
Companies
Contributions from
Research
Institutions
Maintained by
KNIME

Community Contributors
40
Technology
Partners
Distribution
& Consulting
Partners
Community
Contributors
Community
User Base
Academic Institutions:
• Universität Tübingen (BALL, OpenMS)
• Freie Universität Berlin (SeqAn)
• MPI Dresden (ImgLib)
• Universität Dresden (Palladin)
• ETH Zürich (OpenBIS)
• Dublin University (OMERO)
• University of Wisconsin (ImageJ2)
• …
Commercial Contributors:
• Dymatrix Consulting Group (Uplift Nodes)
• Eli Lilly (ChemInf suite)
• Novartis (RDKit, Indigo)
• Vernalis (Proteomics)
• Cenix (REST Nodes)
• Böhringer-Ingelheim (various sponsored nodes)
• …

Bioinformatics
https://tech.knime.org/bioinformatics-and-next-
generation-sequencing-extensions

OpenMS
Open-source software C++ library for
liquid chromatography–mass
spectrometry data management and
analyses.
https://tech.knime.org/community/bio
inf/openms

SeqAn
Open-source C++ library of efficient
algorithms and data structures for the
analysis of sequences with the focus
on biological data.
https://tech.knime.org/seqan-nodes-
for-knime

NGS
Nodes and workflows used for
processing next generation
sequencing results
https://tech.knime.org/community/n
ext-generationsequencing

knime4bio
Set of custom nodes for analysing NGS
data
https://code.google.com/p/knime4bio/

Image Processing
https://tech.knime.org/community/image-processing

Active Classification in Cell Assay Images
• Different modules for segmentation and
feature extraction
• Active Learning

Active Classification in Cell Assay Images
CellMiner Nodes
Plate/Image Reading
– Plate Reader, Plate Editor, Plate View
Preprocessing
– Noise Filtering, Lowpass Filter
Segmentation
– Threshold based Segmentation,
Voronoi Segmentation
Features
– Line, Histogram, Texture, RGB, Zernike Moments, Shape
Active Classification

Chemistry and Cheminformatics
https://tech.knime.org/cheminformatics-extensions

Selected Open Source extensions
50

Selected commercial extensions
51

Overview of types in KNIME
52
• Basic KNIME types
• string, integer, double
• KNIME core chemistry types:
• smiles, sdf, mol, mol2
• Structures in these formats can
be rendered in KNIME tables

Nodes for type manipulation
53
• Molecule Type Cast
• Casts any string as a chemical type (i.e. It
tells KNIME “This is a smiles string”)
• Useful when reading data form a csv file or
database.
• Marvin MolConverter
• Provided by Chemaxon/Infocom
• Translates seamlessly between types
(smiles  sdf  mrv)

Nodes for reading and writing files
54
Reader and writers provided for:
- sdf, smiles, mol, mol2

Sketching chemical structures – use Marvin
55
MarvinSketch
• Provided by Chemaxon/Infocom
• Sketch structures in the configuration dialog
• Execute node to inject structures into workflow

RDKit
56
• Open source cheminfo library in c++
• Wrappers for KNIME maintained by the open
source community
• Useful for:
Descriptor calculation
Cleaning structures
InChi conversion
Standardizing smiles
Fingerprints
Scaffolds/substructures
Reaction simulation
and more…

Infocom JChem KNIME Nodes
Extensions of ChemAxon’s tool for
KNIME workflow
Infocom implements it with the
support of ChemAxon
Contains over 90% of ChemAxon's
cheminformatics functionality

ChEMBL
58
A public database of bioactive druglike compounds
~1.3 mio compounds
~ 9k targets
~12 mio bioactivitities
Provided by the European Bioinformatics Institute
Accessible online at www.ebi.ac.uk/chembl
or via EBI provided KNIME nodes…

New Node: ChEMBLdb Connector
59
Access data in ChEMBL via a web service call
(internet access required)
Lookup by ChEMBLID or InChi Key
Retrieve structure and bioactivity data
Compound search using smiles
exact, similarity, or substructure

Tool Integrations

• Select the KNIME version for your computer
– (Mac, Win, Linux)
• Copy to your local machine
• Unpack the file in a “nice” place
Install KNIME
61

Start KNIME
Go to the installation directory and launch KNIME.
62

The Workspace
• The workspace is the folder in which workflows (and
potentially data files) for the current KNIME session is
stored.
• Workspaces are portable (just like KNIME)
63

Starting KNIME for the first time
64
Install additional
extensions
Goes straight to the
KNIME workbench

The KNIME Workbench
65

A basic workflow
66

More on nodes…
A node can have 3 states:
67
Idle:
The node is not yet configured and can not
be executed with it’s current settings.
Configured:
The node has been set up correctly, and may
be executed at any time
Executed:
The node has been successfully executed. Results
may be viewed and used in downstream nodes.

Node configuration
• Most nodes require configuration
• To access a node configuration window:
• Double-click the node
• Right-click > Configure
68

Node execution
• Right-click node
• Select Execute in context menu
• If execution is successful, status shows
green light
• If execution encounters errors, status
shows red light
69

Node views
• Right-click node
• Select Views in context menu
• Select output port to inspect execution
results
70

Hotkeys (for future reference)
71

A Peak under the Hood:
KNIME (Node) Development
72

Node Architecture
• KNIME interacts only with
a Node
• Node takes care of
embedding the node in
the infrastructure
• New nodes implement
Model/View/Dialog
73
class Node
(final)
class
Node-
Dialog-
Pane
(abstract)
class
Node-
View
(abstract)
class
Node-
Model
(abstract)
class NodeFactory (abstract)

Node Extension Wizard
• Included in the KNIME Developer Version
• Allows creation of plugin projects including
functioning KNIME nodes (with sample code)
• Helpful to easily create all node classes
– Generates all Java classes
– Node is registered with the plugin project
– Launch KNIME and enjoy the new node working!
74

75

• Specify all settings to
create a new KNIME node
– In a completely new plugin
project, or
– Into an existing project
• Node type: Sink, Source,
Learner, Predictor,
Manipulator, Visualizer,
Meta, or Other
• Include sample code or
not
76

• Contains all Java
classes (including
sample code)
• Node is registered in
the plugin.xml
• NodeDialog and
NodeView class are
also created and
registered to the
NodeFactory
77

Node Development
78

Resources
• KNIME pages (www.knime.org)
• APPLICATIONS for example workflows
• LEARNING HUB under RESOURCES
www.knime.org/learning-hub
• KNIME Tech pages (tech.knime.org)
• FORUM for questions and answers
• DOCUMENTATION for documentation, FAQ, changelogs, ...
• LABS where to find new experimental nodes
• COMMUNITY CONTRIBUTIONS for development instructions
and third party nodes
• KNIME TV channel on
• KNIME on @KNIME
79

Knime & bioinformatics

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Knime & bioinformatics (20)

More from BioinformaticsInstitute (20)

Recently uploaded (20)