The science behind predictive analytics a text mining perspective

The Science behind
Predictive Analytics:
a Text Mining Perspective

Ankur Pandey
[24]7 Innovation Labs

Predictive Analytics

• Use of statistical techniques to analyse
current data to make predictions about future
events.

• In most cases, we deal with unstructured text
data.

• Text data is very high dimensional & sparse.

Text Data: Representation
• As a pre-processing step, typically:
 Stop-words removal
 Stemming

• Bag of words model
 As an e.g., in Vector Space model, each document in
a corpus is represented by a vector which represents
words as elements. Measures of elements could be:
Term Frequency
TF.IDF
 Most often we uses cosine between vectors to
measure similarity of documents.

Vector Space Model

A term-document matrix

Text Data: Representation
• String models
 e.g., n-gram model

• Semantic models
 Parsing

 Dimensionality Reduction
 e.g., LSA(I) uses a transformation of term-document matrix to
another matrix with reduced singular values.

 Topic Models
 e.g., LDA assumes a distribution of latent topics in each
document.

Latent Dirichlet Allocation
• Assume that number of words in the documents
(in a corpus) follow some distribution.
• Assume that each document is some mixture
(more accurately, distribution) of k abstract
topics.
• Assume that each topic is some mixture (more
accurately, distribution) of words.

• LDA then tries to backtrack from the documents
to find a set of topics that are likely to have
generated the collection.

Text Data: Clustering
• To group similar documents together, in an
unsupervised manner.

• Topic Modelling and several Dimensionality
Reduction methods are inherently clustering
approaches.

• Other important approaches are: k-means
clustering, and hierarchical clustering.

k-means Clustering
• We select k points and group all other points as
neighbours of one of these k points based on
(cosine) similarity.
• Original k points are replaced by centroids of
these groups (clusters), and same process is
repeated until convergence.

Text Data: Classification
• Classification problem is to assign one of the
existing labels to a document in the corpus.

• We assume that we have a given collection of
training data.

Rule-based Classifier
• The data space is modelled with a set of rules. The
rule can be employed over presence (or absence)
of words (or phrases).

• The rules can be further enriched by length of
neighbourhood, position of words (or phrases)
etc.

• Decision tree learning is a kind of rule-based
classification.

Decision Tree Learning

Decision tree showing survival of passengers on the Titanic
(‘sibsp’ is the number of spouses or siblings aboard). The fig. under
the leaves show the probability of survival and the percentage of
observations in the leaf.

Linear Classifier
• Linear Classifiers are those for which the output is
defined to be f= A.D+ b, where D is a document
vector, A is a coefficient vector of same
dimensionality, and b is a scalar.

• Logistic Regression is an e.g., in which the output
f is related to the probability of D being in some
class C (i.e., p(C|D)) via a log function.

References
• Mining Text Data, edited by Charu C. Aggarwal &
ChengXiang Zhai.

• Introduction to Information Retrieval, by
Christopher D. Manning. Prabhakar Raghavan &
Hinrich Schütze.

• Pattern Recognition and Machine Learning, by
Christopher M. Bishop

The science behind predictive analytics a text mining perspective

More Related Content

What's hot (19)

Viewers also liked (6)

Similar to The science behind predictive analytics a text mining perspective (20)

Recently uploaded (20)

The science behind predictive analytics a text mining perspective