A Novel approach for Document Clustering using Concept Extraction

International Journal of Innovative Research in Advanced Engineering (IJIRAE) ISSN: 2349-2763
Issue 05, Volume 3 (May 2016) www.ijirae.com
_________________________________________________________________________________________________
IJIRAE: Impact Factor Value – SJIF: Innospace, Morocco (2015): 3.361 | PIF: 2.469 | Jour Info: 4.085 |
Index Copernicus 2014 = 6.57
© 2014- 16, IJIRAE- All Rights Reserved Page -59
A Novel approach for Document Clustering using Concept
Extraction
Kasula Chaithanya Pramodh Dr. P. Vijayapal Reddy
Computer Science, G.R.I.E.T, India Computer Science, G.R.I.E.T, India
Abstract— In this paper we present a novel approach to extract the concept from a document and cluster such set of
documents depending on the concept extracted from each of them. We transform the corpus into vector space by using
term frequency–inverse document frequency then calculate the cosine distance between each document, followed by
clustering them using K means algorithm. We also use multidimensional scaling to reduce the dimensionality within
the corpus. It results in the grouping of documents which are most similar to each other with respect to their content
and the genre.
Keywords— Concept extraction, document clustering, multidimensional scaling, genre detection, K means algorithm
I. INTRODUCTION
According to Wai-chiu Wong et al. [1], traditionally, the clustering of documents was carried out manually. But, with the
advent of machine learning and various text classification algorithms it was possible for the computers to take upon this
task. These algorithms use training datasets for leaning. Hence, the results of these algorithms would be strongly based
on the input datasets provided to it and is not always highly reliable as there are many new terms and concepts which are
born every day. It would be very difficult to keep a check on them and have experts identified training examples for each
and every text class thus generated and to learn a classifier for it in the above manner. Various clustering techniques have
been employed to make this process automatic. The appealing characteristic of cluster analysis is that we can find
clusters directly from the given data without relying on any pre-determined information.
In the vector space model, a document space can be viewed as a vector space composed by a set of orthogonal vectors. In
addition, it is assumed that feature items are linearly independent so that a document can be represented by the feature
vector. The features will be further mapped to the semantic level, improving the efficiency of feature extraction and
accuracy of text representation. After extracting the feature vectors of the document, the clustering techniques are applied
on them. The result is a set of clusters in which the documents belonging to a cluster are highly similar to each other in
terms of their content. All such clusters with the documents are visualized and then reported to the user.
II. RELATED WORK
According to N. Elavarasan et al. [3], Mohammad et al. [2] analyzed and discovered that the existing feature extraction
methods were unable to find the important features and so they developed a multi-level method to overcome this problem.
A binary n-gram method was used in the first level, followed by a statistical method, in order to extract the most relevant
features.
As per Hua-Yan Wang et al. [4], a new approach was designed by for extracting the efficient features from
compositional data. This approach works by finding the optimal projection which maximizes the Dirichlet precision on a
family of DR projections that preserve all the relevant constraints. In reference to Veerabhadrappa and Lalitha
Rangarajan [5], have designed a hybrid method using a multi-level process to extract the important features. A statistical
method was used in the first level to extract the best features followed by analyzing the quality of these individual
features in the second level. The best ones were extracted according to the quality measure.
Hoang Vu Nguyen et al. [6] developed a new method called Dimensionality Reduction/Feature Extraction for OUTlier
Detection (DROUT). Outlier data was the primary focus to extract the most relevant features. Further weighted adjusted
scalar matrix was used to extract the efficient features. Kea is an algorithm that is used for automatically extracting key
phrases from the text. Firstly, it identifies the key phrases using lexical methods, calculates feature values for each
candidate and uses Bayesian supervised learning approach to build a model out of training data with known key phrases.
It then uses this trained model to find key phrases in new documents. According to M. Hammouda et al. [7], Mani and
Bloedorn [23] suggested a method for summarizing multiple documents, depending on the graph representation based on
concepts in the text. Also, another system for topic identification is TopCat. It uses a series of natural language
processing, analyzing frequent item sets and clustering steps to identify topics in a document collection. As per M.
Hammouda et al. [7], Extractor is aimed to achieve a human-like performance. It uses machine learning to extract key
phrases by employing a genetic algorithm to tune its parameters. The extracted key phrases are matched against the
human generated key phases and are evaluated accordingly.

_________________________________________________________________________________________________
III. PROPOSED MODEL
Fig. 3.1 The proposed model for concept extraction and clustering.
The model illustrated in fig 3.1 consists of eight different steps. The preliminary stages involve removing the stop words,
followed by stemming and tokenizing the document. The next stages consist of constructing a tf-idf matrix, calculating
the cosine similarities and cosine distances between the documents. The final level of execution involves clustering and
multidimensional scaling to reduce the dimensionality within the corpus. The last stage is the visualization of the
generated document clusters.
IV.IMPLEMENTATION
A. Corpus description
The corpus used in this work, consists of diverse articles extracted from ‘pocket’. Pocket, previously known as Read It
Later, is an application and service for managing a reading list of articles from the Internet. It is available for OSX,
Windows, iOS, Android, Windows Phone, BlackBerry and web browsers.
This application allows the user to save an article or web page to the cloud for later reading. The article is then sent to
the user's Pocket list (synced to all of their devices) for offline reading. Pocket removes clutter from articles and allows
the user to adjust text settings for easier reading. The application has 17 million users [19] and 1 billion saves (both as of
September 2015) [20]. Some applications, such as Twitter, Flipboard, and Google Currents use Pocket's API [21]. Pocket
was listed among Time's 50 Best Android Applications for 2013 [22]. A collection of 500 articles from various genres
were extracted from Pocket. The concept was extracted from these documents and they have been clustered into groups
such that the documents belonging to the same cluster belong to the same genre.
B. Removal of Stopwords
A large number of words make up a document but only a few of them make a significant contribution to it. Words like
IT, AND, TO, ARE, THE can be found in almost every sentence of the English language. These words are called as stop
words and make up a large fraction of the text in most of the documents. According to W. Francis et al. [8],the ten most
frequently occurring words in English account to 20 to 30percent of tokenizing a document. According to Rachel et al.
[9], they have a very less significance in terms of Information Retrieval (IR) and are therefore called as stopwords, noise
words or the negative dictionary.
So, it is usually worthwhile to remove or ignore all stopwords when performing analysis or processing queries over
text documents. In the first phase of the execution, all such stop words in the English language are removed. NLTK’s list
of stop words are used to identify and separate them from the corpus.
C. Stemming
Morphology is the identification, analysis and description of the structure of a given language's morphemes and other
linguistic units, such as root words, affixes, parts of speech, intonations and stresses, or implied context. As stated by
Mirco et al. [10], it is one of the characteristics of text mining that has to be taken into account while performing text
analysis. Considering the set of words, democracy, democratic and democratization, it is observed that the words,
democratization, democratic are generated by adding a suffix for the word ‘democra’ which is called the stem. Hence, all
such words deriving from the same stem ‘democra’ can be represented as democra*, where the symbol * denotes a
variable length don’t-care match. This has led to the development of conflation techniques which permit the matching of
different forms of the same word.
In this project, one of the most reliable and highly effective stemmer called the snowball stemmer is used to perform
stemming. Snowball is a small string processing language designed for creating stemming algorithms for use in
Information Retrieval. The English snowball stemmer breaks down any word from the English vocabulary to its root
word or the stem.
Remove Stopwords Stemming and
Tokenizing
Creating a data
frame
Constructing a
tf- idf matix
Calculating cosine
similarity and
distance
Clustering using K
means algorithm
Multi- dimensional
Scaling
Visualizing
document clusters

_________________________________________________________________________________________________
D. Lemmatization and Tokenization
The goal of both stemming and lemmatization is to reduce the declensional forms and derivationally limited forms.
According to Cristopher, Prabhakar Raghavan and Hinrich Schutze [11], Stemming refers to the process of cutting down
the ends of the words which often includes the removal of derivational affixes. Lemmatization uses vocabulary and
morphological analysis of words to remove inflectional endings only and return the base or the dictionary form of a word
called as lemma. For example, when referring to a word saw, stemming might just return s, whereas lemmatization would
return either see or saw depending on whether it was used as a verb or noun in the context.In this work, two functions are
defined. One function tokenizes the whole document and stems each and every token thus generated while the other
function only tokenizes the document. This is to filter out all numbers, byte order marks if any, present in the corpus.
E. Data frame construction
Data Frame is a two dimensional labeled data structure with columns of potentially different types. In this structure,
each column contains measurements of one variable and each row contains one case. It accepts many kinds of inputs like
lists, dictionaries, series, 2D numpy. ndarray, structured or record ND array, a series, or any other data frame.In this
module the pandas data frame data structure is implemented. It is a two-dimensional, size-mutable, potentially
heterogeneous tabular data structure with labeled axes (rows and columns). Arithmetic operations align on both row and
column labels. It can be considered as a dictionary-like container for series objects. A DataFrame with stemmed
vocabulary as index and tokenized words as the column is created. The advantage of it is that any stem word can be
looked up to return a full token.
F. Term frequency- Inverse document frequency (tf- idf)
Tf- idf stands for term frequency-inverse document frequency. In text mining and information retrieval, tf-idf weight
has much importance. This weight tells us how important a word is to a document in a collection or corpus. This
importance is directly proportional to the number of times a word appears in a document. Inherently, if a word appears
frequently in a document, it’s important, so the word is given a high score. But, if the same word appears in many
documents, it’s not a unique identifier. So, the word is given a low score. The tf-idf weight is made of two terms, Term
Frequency (TF) and Inverse Document Frequency (IDF). Term frequency can be calculated by dividing the number of
times a word appears in a document by the total number of words in that document. Whereas, the Inverse Document
Frequency can be computed as the logarithm of the number of the documents in the corpus, divided by the number of
documents where the specific term appears.
With N: total number of documents in the corpus N=|D| and is the number of documents where the
term t appears (i.e., tf(t, D) ). If the term is not in the corpus, this will lead to a division-by-zero. It is therefore
common to adjust the denominator to 1+ .
Then tf–idf can be calculated as: tf-idf (t,d, D) = tf (t,d) idf (t, D)
A high weight in tf–idf is reached by a high term frequency (in the given document) and a low document frequency of
the term in the whole collection of documents; the weights hence tend to filter out common terms. Since the ratio inside
the idf's log function is always greater than or equal to 1, the value of idf (and tf-idf) is greater than or equal to zero. As a
term appears in more documents, the ratio inside the logarithm approaches 1, bringing the idf and tf-idf closer to 0. In
this method, the term frequency-inverse document frequency (tf-idf) vectorizer parameters are defined and then
the document content list is converted into a tf-idf matrix. To get this matrix, the word occurrences by document are
counted and transformed into a document-term matrix (dtm). It’s also called a term frequency matrix. The term
frequency inverse document frequency weighting is computed and three parameters are defined. They are df_max,
idf_min and ngram_range. df_max is the maximum frequency within the documents a given feature can have, which isto
be used in the tf-idf matrix. Idf_min is an integer then the term would have to be in at least the integer number of
documents specified in order to be considered. ngram_range is the appropriate range of ngrams which is user defined
depending on the corpus.
G. Cosine similarity
Cosine similarity is a measure of similarity between two vectors of an inner product space that measures the cosine of
the angle between them. This metric can be considered as a measurement of orientation and not magnitude.

_________________________________________________________________________________________________
It can be seen as a comparison between documents on a normalized space because, not only the magnitude of each
word count (tf-idf) of each document is considered but also the angle between the documents. Singhal et al. [12] explain
how cosine similarity is a useful measure to know about how similar two documents are likely to be in terms of their
subject matter.
The cosine of two vectors can be derived by using the Euclidean dot product formula:
Given two vectors of attributes A and B, the cosine similarity, cos ), is represented using a dot product and
magnitude as,
where, Ai and Bi are components of vector A and B respectively. The resulting similarity ranges from -1 meaning
exactly opposite, to 1 meaning exactly the same, with 0 indicating orthogonality, and in between values indicating
intermediate similarity or dissimilarity. In this section the cosine similarity against the tf-idf matrix is measured. Cosine
distance is calculated as one minus the cosine similarity of each document. We will further use this distance for plotting a
Euclidian plane.
H. K means clustering
K-means clustering as sited in, MacQueen, 1967[13], is a method used to partition a data set into K groups
automatically. Initially K clusters are selected and they are iteratively refined through the process in the following
manner.
1) Firstly, the closest cluster center for a point is identified and this point is allocated to it.
2) Each cluster center Cj is updated to be the mean of its constituent points [14].
From the mathematical perspective, according to Alizade et al. [15], given a data set X = {x1, x2, … , xN} in ℝD, i.e. N
points (vectors) each with D attributes (components), K-means algorithm divides X into K exhaustive and mutually
exclusive clusters P = {p1, p2, … , pK },⋃ = =1 , ⋂ = ∅ for 1 ≤ i ≠ j ≤ K. This algorithm generates clusters by
optimizing a criterion function. As stated by Celebi et al., 2013 [16], the most intuitive and frequently used criterion
function is the Sum of Squared Error (SSE) given by:
Where ||∙||2 , denotes the Euclidian (ℒ2) norm and
This is the centroid of cluster whose cardinality is | |.The optimization of SSE is often referred to as the minimum
SSE clustering (MSSC) problem. In this module, appropriate integer number of predetermined clusters are initialized.
Then, each observation is assigned to a cluster so as to minimize the minimum sum of squares error. The mean of the
clustered observations is calculated and used as the new cluster centroid. These observations are reassigned to clusters
and centroids are recalculated in an iterative process until the algorithm reaches the convergence. It needs to be run
several times for the algorithm to converge to a global optimum as K means is susceptible to reaching local optima. A
dictionary of ‘titles’ is created which contains the title of the document, ‘synopses’, which contains the actual content of
the document and ‘clusters’ which is presently empty, would later be filled with the number of the cluster to which the
document belongs. A data frame with a list named ‘clusters’ as index and ‘titles’, ‘synopses’ as the columns is created.
After this, the top n terms for each synopses that are nearest to the cluster centroid are identified. This gives a good sense
of the main topic of the cluster.
I. Multidimensional Scaling
Multidimensional Scaling is a means to visualize the level of similarity of individual cases in the dataset. It is
particularly used to display the information contained in a distance matrix. It uses a set of related ordination techniques
popular in information visualization.
According to, Borg et al., 2005[17], an MDS algorithm places each object in N-dimensional space such that the
distances between the objects are preserved as much as possible. Each object is then assigned coordinates in each of
the N dimensions. There may be more than 2 dimensions in an MDS plot and it is specified by the priori. Choosing N=2
optimizes the object locations for a two-dimensional scatter plot.

_________________________________________________________________________________________________
Fig. 4.2 A sample output showing the generated document clusters.
J. Principal Coordinates Analysis
It is also known as Torgerson Scaling or Torgerson–Gower scaling. A matrix containing the dissimilarities between
pairs of items is taken as an input and the coordinate matrix whose configuration minimizes a loss function, also referred
to as strain, is given as an output.
For example, consider a matrix D = [dij], consists of aerial distances between many cities, where, dij is the distance
between the coordinates of ith
and jth
city which can be calculated as,
Now, the coordinates of the cities are to be found. This problem is addressed in classical MDS.
General forms of loss functions called Stress in distance MDS and Strain in classical MDS. The strain is given by:
The distance matrix is converted into a two dimensional array using multidimensional scaling. Principal component
analysis can also be used to achieve this.
K. Visualizing document clusters
Matplotlib is a python 2D plotting library. It produces publication quality figures in a variety of hardcopy formats and
interactive environments across platforms. Matplotlib can be used in python scripts, the python and ipython shell
(alaMATLAB or Mathematica), web application servers, and six graphical user interface toolkits. As per Hunter et al.
[18], Matplotlib tries to make easy things easy and hard things possible. You can generate plots, histograms, power
spectra, bar charts, error charts, scatter plots, et cetera, with just a few lines of code. The process of visualizing the
clusters with matplotlib can be achieved by implementing the below process. Firstly, a dataframe that comprises of the
result of the data frame which was produced in the previous module with the cluster members and also the titles is
created. A plot is set up, the necessary margins are added and scale is set. Now, iteration is done through groups to layer
the plot. The cluster category is matched with the document name giving it a specific color and a position in the plot. The
following is a sample output. The documents belonging to a cluster are represented with the same color. Thus, in the fig
4.1, there are five clusters which are formed from the given set of documents. The documents belonging to the same
cluster have the same concept and belong to the same genre.

_________________________________________________________________________________________________
V. RESULTS AND DISCUSSION
The aim of the project was to find the features from the documents belonging to different domains and classify them
according to their respective genres without using any kind of training datasets and training algorithms but solely
depending on the linguistics of the language. Considering a few articles from the corpus, the different elements of the
articles belonging to the same cluster are observed and checked for whether the developed method was good enough to
achieve its purpose. Upon analyzing the articles, ‘Why generation Y Yuppies are unhappy’ and ‘Children today are
suffering a severe deficit of play’, it is observed that, both of these articles talk about happiness, reality and expectations.
They tell the reader about how the change has gone from the old to the new, what the generations have lost during this
process and about how psychologically these changes affected them. Both of them talk about concepts like depression,
education, ability, anxiety, young people, age group and unhappiness in common.
Now considering the cluster containing the articles, ‘The age of the introvert entrepreneur’ and ‘The Buffet formula’,
they speak about people, technology, company, energy, learning and education in common. Although, on analyzing the
cluster containing the articles, ‘Product strategy means saying no’ and ‘Diamonds are bullshit’, there seems to be a very
less probability of them belonging to the same cluster. The article, ’Diamonds are bullshit’, illustrates on how the value
of diamonds varies with time by discussing concepts of intrinsic value and market manipulation. While the later,
‘Product strategy means saying no’ tells about how features of an application would influence your audience along its
lifetime, how adding small or big features include large hidden costs that reveal over time and also how to say ‘no’ to
adding something that isn’t needed to your application. After performing the tests, the Accuracy of the model is
computed as 81.8%, Precision is as high as 80%, Recall is calculated and is found to be 81%. The F- measure is found to
be 80.38%.
TABLE 5.1
S.NO EVALUATION MEASURE RESULT
1 ACCURACY 81.81%
2 PRECISION 80%
3 RECALL 81%
4 F- MEASURE 80.38%
VI. CONCLUSIONS AND FUTURE WORK
In this paper, an approach to classify the documents according to their genres by automatically extracting the feature
from them was successfully developed. This removes the complexities involved in training algorithms and training data.
There are many challenges and future developments possible to solve the problems of poor spellings, punctuation and
grammar, incomplete sentences which would have a greater impact if they are important or significant for a document. It
can be used to look in for identifying and analyzing trends in documents collected over a period of time. Attribute
analysis can also be performed. Given a set of documents we can identify relationships between attributes (features that
have been extracted from the documents) such as the presence of one pattern implies the presence of another pattern.
Applications include advanced E-mail spam filtering where unwanted and unsolicited emails are detected and prevented
from getting into the user’s inbox, categorizing newspaper articles and newswires into topics, organizing web pages into
hierarchical categories, analyzing and assessing various symptoms and complexities of different diseases by examining
patients’ records, sorting journals and abstracts by subject categories.
REFERENCES
[1] Wai-chiu Wong and Ada Wai-chee Fu,”Incremental Document Clustering for Web Page Classification”,
Department of Computer Science and Engineering, Chinese University of Hong Kong, Hong Kong.
[2] Mohammad M. Masud, Latifur Khan, Bhavani Thuraisingham, A scalable multi-level feature extraction technique
to detect malicious executable, Lecture Notes in Computer Science (Springer),Volume 10, 2008, pp. 33-45.
[3] Elavarasan, Dr. K.Mani, “A Survey on Feature Extraction Techniques”, Innovative Research in Computer and
Communication Engineering, Vol. 3, Issue 1, January 2015.
[4] Hua-Yan Wang, Qiang Yang,Hong Qin, Hongbin Zha, “Dirichlet Component Analysis: Feature Extraction for
Compositional Data”, International Conference on Machine Learning, 2008, pp.20-28.
[5] Veerabhadrappa, Lalitha Rangarajan, “Multi-Level Dimensionality Reduction Methods Using Feature Selection and
Feature Extraction”, International Journal of Artificial Intelligence & Applications, Volume 1, Number 4, 2010, pp.
54-68.

_________________________________________________________________________________________________
[6] Hoang Vu Nguyen, Vivekanand Gopalkrishnan, “Feature Extraction for Outlier Detection in High-Dimensional
Spaces”, Journal of Machine Learning Research, Volume 10, Issue 2, 2010, pp. 252-262.
[7] Khaled M. Hammouda , Diego N. Matute, and Mohamed S. Kamel, “CorePhrase: Keyphrase Extraction for
Document Clustering”.
[8] W. Francis.” Frequency Analysis of English Usage: Lexicon and Grammar”. Houghton Mifflin, 1982.
[9] Rachel Tsz-Wai Lo, Ben He, Iadh Ounis, “Automatically Building a Stopword List for an Information Retrieval
System”, Department of Computing Science University of Glasgow 17 Lilybank Gardens Glasgow, UK.
[10] Mirco Popovic,Peter Willet, “The Effectiveness of Stemming for Natural- Language Access to Solvene Textual
Data”,Journal of the American Society for Information Science, v43 n5 p384-90 Jun 1992.
[11] Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze, “Introduction to Information Retrieval”,
Cambridge University Press. 2008.
[12] Singhal, Amit (2001). "Modern Information Retrieval: A Brief Overview". Bulletin of the IEEE Computer Society
Technical Committee on Data Engineering 24 (4): 35–43.
[13] J. B. MacQueen, Some Methods for classification and Analysis of Multivariate Observations, Proceedings of 5-th
Berkeley Symposium on Mathematical Statistics and Probability, Berkeley, University of California Press, 1:281-
297., 1967.
[14] Wagstaff, K., Cardie, C., Rogers, S., Schrödl, S., 2001. “Constrained k-means clustering with background
knowledge”, ICML, 577-584.
[15] A. Alizade Naeinia, A. Jamshidzadeh b, *, M. Saadatseresht a, S. Homayouni c,” An Efficient Initialization Method
for K-means Clustering of Hyperspectral Data”.
[16] Celebi, M.E., Kingravi, H.A., Vela, P.A., 2013. “A comparative study of efficient initialization methods for the k-
means clustering algorithm”. Expert Systems with Applications 40, 200- 210.
[17] Borg, I., Groenen, P. (2005). “Modern Multidimensional Scaling: theory and applications” (2nd ed.). New York:
Springer-Verlag. pp. 207–212. ISBN 0-387-94845-7.
[18] Hunter, J. D,”Matplotlib: A 2D graphics environment”, Computing In Science & Engineering, Vol 9, IEEE
Computer Society, 2007.
[19] “Pocket built a save button for the internet — what's next?" The Verge. September 8, 2015. Retrieved September 9,
2015.
[20] "1 Billion Saves to Pocket!”, Read It Later Inc. May 15, 2014. Retrieved January 19, 2015.
[21] "Pocket Operating System Compatibility". Retrieved July 30, 2013.
[22] Jared Newman (July 1, 2013). "50 Best Android Apps for 2013".Time. Retrieved June 30, 2013.
[23] I. Mani and E. Bloedron, “Multi-document Summarization by Graph Search and Merging”, in Proceedings of the
Fourteenth National Conference on Artificial Intelligence (AAAI - 97), 1997, pp. 622-628.

A Novel approach for Document Clustering using Concept Extraction

More Related Content

What's hot (19)

Viewers also liked (20)

Similar to A Novel approach for Document Clustering using Concept Extraction (20)

More from AM Publications (20)

Recently uploaded (20)

A Novel approach for Document Clustering using Concept Extraction