A machine learning approach to web page filtering using ...

Available online at www.sciencedirect.com

Decision Support Systems 44 (2008) 482 – 494
www.elsevier.com/locate/dss

A machine learning approach to web page filtering
using content and structure analysis
Michael Chau a,⁎, Hsinchun Chen b
a
School of Business, The University of Hong Kong, Pokfulam, Hong Kong
b
Department of Management Information Systems, The University of Arizona, Tucson, Arizona 85721, USA
Received 15 January 2006; received in revised form 10 February 2007; accepted 13 June 2007
Available online 22 June 2007

Abstract

As the Web continues to grow, it has become increasingly difficult to search for relevant information using traditional search
engines. Topic-specific search engines provide an alternative way to support efficient information retrieval on the Web by providing
more precise and customized searching in various domains. However, developers of topic-specific search engines need to address two
issues: how to locate relevant documents (URLs) on the Web and how to filter out irrelevant documents from a set of documents
collected from the Web. This paper reports our research in addressing the second issue. We propose a machine-learning-based
approach that combines Web content analysis and Web structure analysis. We represent each Web page by a set of content-based and
link-based features, which can be used as the input for various machine learning algorithms. The proposed approach was implemented
using both a feedforward/backpropagation neural network and a support vector machine. Two experiments were designed and
conducted to compare the proposed Web-feature approach with two existing Web page filtering methods — a keyword-based
approach and a lexicon-based approach. The experimental results showed that the proposed approach in general performed better than
the benchmark approaches, especially when the number of training documents was small. The proposed approaches can be applied in
topic-specific search engine development and other Web applications such as Web content management.
© 2007 Elsevier B.V. All rights reserved.

Keywords: Web page classification; Link analysis; Machine learning; Web mining

1. Introduction has exceeded eight billion, it has become more difficult
for search engines to keep an up-to-date and compre-
The most popular way to look for information on the hensive search index. Users often find it difficult to
Web is to use Web search engines such as Google (www. search for useful and high-quality information on the
google.com) and AltaVista (www.altavista.com). Many Web using general-purpose search engines, especially
users begin their Web activities by submitting a query to when searching for specific information on a given topic.
a search engine. However, as the size of the Web is still Many vertical search engines, or topic-specific
growing and the number of indexable pages on the Web search engines, have been built to facilitate more ef-
ficient searching in various domains. These search
⁎ Corresponding author. engines alleviate the information overload problem to
E-mail addresses: mchau@business.hku.hk (M. Chau), some extent by providing more precise results and more
hchen@eller.arizona.edu (H. Chen). customized features [11]. For example, LawCrawler
0167-9236/$ - see front matter © 2007 Elsevier B.V. All rights reserved.
doi:10.1016/j.dss.2007.06.002

M. Chau, H. Chen / Decision Support Systems 44 (2008) 482–494 483

(www.lawcrawler.com) allows users to search for legal protocols [9,13,15]. These spiders use different algo-
information and provides links to lawyers and legal rithms to control their search. To address the first problem
information and to relevant government Web sites. mentioned above, the following methods have been used
BuildingOnline (www.buildingonline.com) is a special- to locate Web pages relevant to a particular domain:
ized search engine for the building industry, where users
can search by manufacturers, architects, associations, • The spiders can be restricted to staying in particular
contractors, etc. BioView.com (www.bioview.com) and Web domains, because many Web domains have
SciSeek (www.sciseek.com) are two other examples specialized contents [36,50]. For example, most Web
that focus on scientific domains. pages within the domain www.toyota.com will be
Although they provide a promising alternative for relevant to automobiles.
users, these vertical search engines are not easy to build. • Some spiders are restricted to collecting only pages at
There are two major challenges to building vertical most a fixed number of links away from the starting
search engines: (1) How to locate relevant documents on URLs or starting domains [36,45]. Assuming that
the Web? (2) How to filter irrelevant documents from a nearer pages have higher chances of being relevant,
collection? This study tries to address the second issue this method prevents spiders from going too “far
and to propose new approaches. The remainder of the away” from the starting domains.
paper is structured as follows. Section 2 reviews existing • More sophisticated spiders use more advanced graph
work on vertical search engine development, text search algorithms that analyze Web pages and
classification, and Web content and structure analysis. hyperlinks to decide what documents should be
In Section 3 we discuss some problems with existing downloaded. Cho et al. [16] proposed a best-first
Web page filtering approaches and pose our research search spider that used PageRank as the ranking
questions. Section 4 describes in detail our proposed heuristic; URLs with a higher PageRank scores will be
approach. Section 5 describes an experiment designed to visited first by the spider. The spider developed by
evaluate our approach and presents experimental results. McCallum et al. [38] used reinforcement learning to
In Section 6, we conclude our paper with some guide their spiders to retrieve research papers from
discussion and suggestions for future research directions. university Web sites. Focused Crawler locates Web
pages relevant to a pre-defined set of topics based on
2. Research background example pages provided by the user. In addition, it also
analyzes the link structures among the Web pages
2.1. Building vertical search engines collected [7]. Context Focused Crawler uses a Naïve
Bayesian classifier to guide the search process [19]. A
A good vertical search engine should contain as many Hopfield Net spider based on spreading activation also
relevant, high-quality pages and as few irrelevant, low- has been proposed [8,10]. Page content scores and link
quality pages as possible. Given the Web's large size and analysis scores are combined to determine which URL
diversity of content, it is not easy to build a comprehen- should be visited next by the spider. The spider was
sive and relevant collection for a vertical search engine. compared with a breadth-first search spider and a best-
There are two main problems: first search spider using PageRank as the heuristics,
and the evaluation results showed that the Hopfield
• The search engine needs to locate the URLs that Net spider performed better than the other two.
point to relevant Web pages. To improve efficiency, it
is necessary for the page collection system to predict While these methods have different levels of perfor-
which URL is the most likely to point to relevant mance in efficiency and effectiveness, in most cases the
material and thus should be fetched first. resulting collection is still noisy and needs further
• After the Web pages have been collected, the search processing. Filtering programs are needed to eliminate
engine system needs to determine the content and irrelevant and low-quality pages from the collection to be
quality of the collection in order to avoid irrelevant or used in a vertical search engine. The filtering techniques
low-quality pages. used can be classified into the following four categories:

Search engines usually use spiders (also referred to as • Domain experts manually determine the relevance of
Web robots, crawlers, worms, or wanderers) as the each Web page (e.g., Yahoo) [30].
software to retrieve pages from the Web by recursively • In the simplest automatic procedure, the relevance of
following URL links in pages using standard HTTP a Web page can be determined by the occurrences of

484 M. Chau, H. Chen / Decision Support Systems 44 (2008) 482–494

particular keywords [16]. Web pages are considered In addition to general text documents, classification
relevant if they contain the specified keyword, and of Web pages also has been studied. Web pages are often
are considered irrelevant otherwise. noisy, but they provide additional information about
• TFIDF (term frequency ⁎ inverse document frequency) each document. For example, terms marked with
is calculated based on a lexicon created by domain- different HTML tags (such as titles or headings) can be
experts. Web pages are then compared with a set of assigned a higher weight than regular text [34]. Terms
relevant documents, and those with a similarity score from neighborhood Web pages also have been used in
above a certain threshold are considered relevant [2]. attempt to improve classification performance. Howev-
• Text classification techniques such as the Naive er, it turns out to worsen performance because there are
Bayesian classifier also have been applied to Web often too many neighbor terms and too many cross-
page filtering [5,38]. linkages between different classes [5,52]. Use of other
information about neighborhood Web pages has been
To our surprise, it appears that some vertical search proposed. Examples of such information include the
engines do not perform filtering; they assume that most predicted category of a page's neighbors [5,40], anchor
pages found in the starting domains (or at a specified text pointing to a page [21], or a page's outgoing links to
depth) are relevant, e.g., NanoSpot (www.nanospot.com) all other documents [25]. It has been shown that using
[11]. such additional information improves classification
results.
2.2. Text classification
2.3. Analysis of Web content and structure
Text classification is the study of classifying textual
documents into predefined categories. The topic has There has been much research on different ways of
been extensively studied at SIGIR conferences and representing and analyzing the content and structure of
evaluated on standard testbeds. There are a number of the Web. In general, they can be classified into two
major approaches. For example, the Naive Bayesian categories: content-based and link-based. The two
method has been widely used [28,33,38]. It uses the joint approaches are discussed in the following.
probabilities of words and categories to estimate the
probability that a given document belongs to each 2.3.1. Content-based approaches
category. Documents with a probability above a certain The actual HTML content of a Web page provides
threshold are considered relevant to that category. much useful information about the page itself. For
The k-nearest neighbor method is another popular example, the body text of a Web page can be analyzed to
approach to text classification. For a given document, the determine whether the page is relevant to a target
k neighbors that are most similar to a given document are domain. Indexing techniques can be used to extract the
first identified [22,37]. The categories of these neighbors key concepts that represent a page. Information extracted
are then used to decide the category of the given from a document using various techniques can be useful
document. A threshold is also used for each category. for text classification [42]. In addition, the relevance of a
Neural network programs, designed to model the page can often be determined by looking at the title.
human neural system and learn patterns by modifying Words and phrases that appear in the title or headings in
the weights among nodes based on learning examples, the HTML structure are usually assigned a higher
also have been applied to text classification. Feedfor- weight. Such weights can be calculated based on the
ward/backpropagation neural network (FF/BP NN) is TFIDF scores discussed earlier.
usually used [31,39,49]. Term frequencies or TFIDF of Domain knowledge also can be incorporated into the
the terms are used as the input to the network. Based on analysis to improve results. Domain knowledge refers to
learning examples, the network can be trained to predict expert knowledge such as domain-specific lexicon or
the category of a document. rules, often obtained from human experts. For example,
Another new technique used in text classification is words in Web pages can be checked against a list of
called support vector machine (SVM), an approach that domain-specific terms. A Web page containing words
tries to find a hyperplane that best separates two classes that are found in the list can be considered more relevant.
[47,48]. Joachims first applied SVM to a text classifi- The URL address of a Web page often contains use-
cation problem [23]. It has been shown that SVM ful information about the page. For example, from the
achieved the best performance among different classi- URL “http://ourworld.compuserve.com/homepages/
fiers on the Reuters-21578 data set [20,44,51]. LungCancer/”, we can tell that it comes from the domain


compuserve.com, and that it is likely to be related to the be calculated iteratively, making it computationally
topic Lung Cancer. We also know that this page comes expensive.
from a .com site, which may be considered less au- Kleinberg [26] proposed a measure called the HITS
thoritative than pages from a .gov site. Some metrics also (Hyperlink-Induced Topic Search) algorithm, which is
consider URLs with fewer slashes to be more useful than similar to PageRank. In the HITS algorithm, authority
those with more slashes [1]. pages are defined as high-quality pages related to a
particular topic or search query. Hub pages are those that
2.3.2. Web structure analysis are not necessarily authorities themselves but that
In recent years, Web link structure has been widely provide pointers to other authority pages. A page to
used to infer important information about pages. which many others point should be a good authority, and
Intuitively, the author of a Web page A places a link to a page that points to many others should be a good hub.
Web page B if he or she believes that B is relevant to A, Based on this intuition, an authority score and a hub
or of good quality. Usually, the larger the number of in- score can be calculated for each Web page as follows:
links, the better a page is considered to be. The rationale X
is that a page referenced by more people is likely to be AuthorityScoreðpÞ ¼ ðHubScoreðqÞÞ
all q linking to p
more important than a page that is seldom referenced.
One can also obtain the anchor text that describes a link. X
HubScoreðpÞ ¼ ðAuthorityScoreðrÞÞ
Anchor text is the clickable text of an outgoing link in a
all r linking to p
Web page. Anchor text may provide a good description
of the target page because it represents how other people A page with a high authority score is one pointed to by
who have linked to the page actually describe it. many hubs, and a page with a high hub score is one that
In addition, it is reasonable to give a link from an points to many authorities. One example that applies the
authoritative source (such as Yahoo) a higher weight HITS algorithm is the Clever search engine [6], which
than a link from an unimportant personal homepage. has achieved a higher user evaluation than the manually
Researchers have developed several methods to incor- compiled directory of Yahoo.
porate this into link analysis. Among these algorithms,
PageRank and HITS are the two most widely used. 3. Research questions
The PageRank algorithm is computed by weighting
each in-link to a page proportionally to the quality of the Based on the review, we identified several problems
page containing the in-link [3]. The quality of these with traditional approaches to Web page filtering. Firstly,
referring pages also is determined by PageRank. Thus, a manual approach is very labor-intensive and time-
the PageRank of a page p is calculated recursively as consuming. Although such approach can achieve high
follows: quality, it is usually not feasible under limited resources.
The keyword-based and the lexicon-based approaches
PageRankðpÞ ¼ ð1 À dÞ þ d can automate the process, but they both have short-
X PageRankðqÞ
Â comings. A simple keyword-based approach cannot deal
all q linking to p
cðqÞ with problem of polysemy, i.e., words having more than
one semantic meaning. For example, a Web page
where d is a damping factor between 0 and 1,c(q) is the containing the word cancer might well be a medical
number of out-going links in q. report about treatment for lung cancer or the horoscope
Intuitively, a Web page has a high PageRank score if for people born under the zodiac sign of cancer. As a
the page is linked from many other pages, and the scores result, this approach can easily fail to eliminate irrelevant
will be even higher if these referring pages are also good pages, thus lowering precision. On the other hand, as
pages (pages that have high PageRank scores). It is also people often use different terms to refer to the same
interesting to note that the PageRank algorithm follows a concept, e.g., lung cancer and lung neoplasm, this
random walk model. The PageRank score of a page is approach also can easily miss out relevant pages, thus
proportional to the probability that a random surfer click- lowering recall. The lexicon-based approach, which used
ing on random links will arrive at that page. Applied in the the TFIDF-based similarity score between each docu-
commercial search engine Google, this score has been ment and a given domain lexicon, alleviates the problem
shown to be very effective for ranking search results [3]. by considering all terms in the documents. However,
Computation time, however, is a main problem in using TFIDF calculation can be biased by the collection; if the
PageRank. The PageRank score of each Web Page has to collection is “noisy”, irrelevant terms can possibly get a


very high IDF score and thus a high TFIDF score. In of words, each Web page is represented by a limited
addition, both the keyword-based and the lexicon-based number of content and link features. This reduces the
approaches do not robustly resist text spamming, a dimensionality (the number of attributes used) of the
popular practice in which Web page authors manipulate classifier and thus the number of training examples
their page content to boost ranking. needed. The characteristics of Web structure also can be
Using text classifiers for Web page filtering seem to be incorporated into these “Web features.”
the most promising approach, given their good perfor- Based on our review of existing literature, we deter-
mance in traditional text classification. However, one mined that in general, the relevance and quality of a Web
problem is that most classifiers were evaluated using at page can be reflected in the following aspects: (1) the
least 2/3 of the data for training in the hold-out sampling content of the page itself, (2) the content of the page's
method. The problem becomes even worse in other neighbor documents, and (3) the page's link information.
evaluation methods such as k-fold cross-validation and Several features are defined for each aspect.
leaving-one-out [27,43], in which (100 − k)% of the data
and all but one instance, respectively, were used for 4.1. Page content
training. It is not feasible to obtain so large a set of training
data in vertical search engine creation because usually The content of a page is probably the primary factor in
only a small number of documents are tagged for determining whether a page is relevant to a given domain.
classifying a large number of documents. It would be As mentioned earlier, we represented the content of each
very expensive and time-consuming to tag a large number page by a set of feature scores rather than a vector of
of documents manually. Also, most existing text classi- words. We adopted an automatic approach that extracted
fication techniques do not make use of domain knowl- all the terms from a page and compared them with a
edge, which is important in vertical search engines. domain lexicon, similarly to the method used in Baujard et
On the other hand, the hyperlink structure of the Web al. [2]. We looked at both the number of relevant terms
has been studied and applied with considerable success that appeared in the page title and the TFIDF scores of the
in Web structure mining research. For example, the terms that appeared in the body of the page. Two feature
PageRank and HITS algorithms have been widely used scores were defined:
in Web search result ranking. These techniques can help
identify Web pages with high quality and relevance 1. Title(p) = Number of terms in the title of page p found
[3,26]. In addition, such techniques can help identify in the domain lexicon
Web pages that are in the same community and thus the 2. TFIDF(p) = Sum of TFIDF of the terms in page p
same domain [29]. These issues are exactly those that found in the domain lexicon.
need to be addressed in Web page filtering applications.
However, the application of these Web structure mining 4.2. Page content of neighbors
techniques in Web page filtering has not been much
investigated. It would be an interesting research question To incorporate the page content of the neighbors of a
to study the effectiveness of using Web structure mining page, a score from each neighborhood document can be
in Web page filtering. used instead of including all the terms from neighbor-
In this study, the following research questions are hood documents, which appears to be more harmful than
investigated: (1) Can Web structure analysis techniques be helpful [5,52]. In our approach, three types of neighbors
used to help create a vertical search engine? (2) were considered: incoming, outgoing, and sibling [5].
Can domain knowledge be used to enhance Web page For any page p, incoming neighbors (parents) are the set
filtering for a vertical search engine? (3) Can Web page of all pages that have a hyperlink pointing to p. Outgoing
classification be applied to a large collection (e.g., a neighbors are pages whose hyperlinks are found in p.
million documents) with only a small number (a few Sibling pages are those pages that are pointed by any of
hundred) of training examples? the parents of p. An example is shown in Fig. 1. In the
example, pages a, b, and c are incoming neighbors of p;
4. A Web-feature approach pages f and g are outgoing neighbors of p, and pages d
and e are siblings of p.
To address the problems with current approaches in Two content scores (title and TFIDF scores) of the
Web page filtering, we propose an approach that in- neighborhood documents were determined similarly to
corporates Web content and structure analysis into Web those created in the previous aspect. Six features were
filtering. Instead of representing each document as a bag used: the averages of the two scores for all incoming


6. Anchor(p) = Number of terms in the anchor texts
describing page p found in the domain lexicon

4.4. FF/BP NN Text classifier

In total, 14 features have been identified and can be
used as the input values to a classifier. We used a neural
network (NN) [12,35] and a support vector machine
Fig. 1. Examples of incoming, outgoing, and sibling pages. (SVM) [47,48] as our classifiers. A feedforward/back-
propagation neural network (FF/BP NN) had been adopted
because of its robustness and wide usage in classification
neighbors, the averages for all outgoing neighbors, and [31,39,49]. The algorithm used is summarized as follows:
the averages for all siblings.
4.4.1. Initializing the network
1. InTitle(p) = Average(number of terms in the title of A neural network was first created with three layers,
page q found in the domain lexicon) for all incoming namely the input layer, the hidden layer, and the output
pages q of p layer. The input layer of the neural network consisted of a
2. InTFIDF(p) = Average(sum of TFIDF of the terms in threshold unit and 14 nodes that corresponded to the
page q found in the domain lexicon) for all incoming 14 feature scores of each page. The output layer
pages q of p consisted of a single output node which determined the
3. OutTitle(p) = Average(number of terms in the title of relevance of a page (whether or not a Web page should be
page r found in the domain lexicon) for all outgoing included in the vertical search engine). The number of
pages r of p nodes in the hidden layer was set at 16 and the learning
4. OutTFIDF(p) = Average(sum of TFIDF of the terms rate at 0.10. These parameters had been set based on
in page r found in the domain lexicon) for all out- some initial experimentation using a small subset of our
going pages r of p data. The parameters that achieved the best performance
5. SiblingTitle(p) = Average(number of terms in the title were used throughout the experiment.
of page s found in the domain lexicon) for all sibling
pages s of p 4.4.2. Training and tuning the network
6. SiblingTFIDF(p) = Average(sum of TFIDF of the The training documents were passed to the network
terms in page s found in the domain lexicon) for all for learning (the method of selecting the training set will
sibling pages s of p be discussed in Section 5.5). The training documents
were then further divided into two sets: 80% of the
4.3. Link analysis documents were used for training and 20% were used for
tuning. The 14 features of each training document, as
Connectivity (link analysis) was used to represent the well as a binary score representing whether the document
quality of a page. Link analysis, such as number of in- was relevant, were presented to the network. Each
links, HITS and PageRank, have been useful in many feature score was normalized to a value between 0 and 1
Web applications such as search result ranking [3,6], but using the sigmoidal function. The network then updated
have not been used in text classification. To incorporate the weights of its connection, based on the training
link analysis scores in our filtering approach, six scores, documents. After all training documents had passed
namely hub score, authority score, PageRank score, through the network once, the tuning documents were
number of in-links, number of out-links, and number of presented to the network and the mean square error
relevant terms in the anchor texts, were used as features. (MSE) of the network was recorded. The whole process
was repeated 3000 times (i.e., 3000 epochs) and the
1. Hub(p) = Hub score of page p calculated by the HITS network with the lowest MSE was selected.
algorithm
2. Authority(p) = Authority score of page p calculated 4.4.3. Testing
by the HITS algorithm Each testing document was presented to the trained
3. PageRank(p) = PageRank score of page p network, which tried to predict whether the document
4. Inlinks(p) = Number of incoming links pointing to p was relevant. The predictions were recorded and used to
5. Outlinks(p) = Number of outgoing links from p calculate the performance measures.


4.5. SVM Text classifier The medical lexicon was created based on the
Metathesaurus, part of the Unified Medical Language
In order to allow for a better comparison, a support System (UMLS) developed by the National Library of
vector machine also was used because of its outstand- Medicine. About 600,000 medical phrases were
ing performance in traditional text classification [51]. It extracted from the Metathesaurus. The lexicon was
performed classification based on the same set of manually edited by a medical librarian and two filtering
feature scores. Our SVM classifier involved the fol- programs were developed and applied to refine the
lowing steps: lexicon. The resulting lexicon has 300,442 unique terms.
To evaluate the proposed Web page classification
4.5.1. Model selection approaches, 1000 documents were randomly chosen
A linear kernel function had been chosen for our from the testbed. Each of these documents was processed
SVM classifier because it is simple, learns quickly, and automatically to calculate its feature scores and keyword
has been shown to achieve performance comparable to vector. All other Web pages in the testbed were
that of non-linear models like polynomial classifiers and accessible for content, neighbor and link analysis during
radial basis functions in text classification applications the process, meaning that such metrics as PageRank and
[20,23]. HITS scores were calculated over the entire set of
documents instead of just the 1000 chosen. Two graduate
4.5.2. Training students with medical training were also recruited to
Each training example was represented as a vector of classify each document manually as either “acceptable”
the 14 features selected and presented to the SVM to or “not acceptable” for a medical search engine.
learn the feature weights.
5.2. Benchmark approaches
4.5.3. Testing
Similarly to the neural network algorithm, the SVM The proposed neural network approach (NN-WEB)
tried to predict, on the basis of its classification model, and support vector machine approach (SVM-WEB) were
whether each document in the testing set was relevant to compared against two benchmark approaches: (1) a
the chosen domain. The results were recorded and used lexicon-based approach (LEXICON), and (2) a keyword-
for evaluation. based support vector machine approach (SVM-WORD).
The lexicon-based approach was chosen because it is fast
5. Evaluation and has been used in various information retrieval appli-
cations. The keyword-based SVM approach was selected
5.1. Experiment testbed because it has been shown to achieve the best perfor-
mance in traditional text classification problems [51].
In order to evaluate the proposed approach, two The lexicon-based approach was adopted from
experiments that compared the proposed approaches Baujard et al. [2]. TFIDF score was calculated for
with traditional approaches were conducted. The those terms found in the medical lexicon. The Jaccard's
medical field was chosen as the domain for evaluation similarity score between every document in the training
because many diverse users (including medical doc- set and the lexicon was calculated. Jaccard's score is one
tors, researchers, librarians and general public) seek of the most widely used similarity scores in information
important and high-quality information on health retrieval [46]. A threshold that divided these training
topics on the Web. It is also important for them to documents into the two classes (relevant and irrelevant)
distinguish between Web pages of good and poor with the highest accuracy was determined. This thresh-
quality [14]. old was then used for testing.
A Web page testbed and a medical lexicon created in The second benchmark approach was a keyword-based
previous research were used [8]. The Web page testbed SVM approach adopted from Joachims's [23]. In the pre-
was built by running a random-first search that started processing stage, each document was first tokenized into
with 5 URLs in the medical domain and traversed the single words. Common functional terms that did not bear a
Web following random outgoing links. The random-first significant semantic meaning (e.g., a, of, and is) then were
search was run until 1 million pages had been collected filtered based on a pre-defined stop-word list. In order to
and indexed. The testbed represented a typical collection reduce the number of unique words and the vector size, we
from simple Web spiders, and consisted of 1,040,388 also followed Joachims's design by applying suffix-
valid, unique Web pages. stripping (stemming) to the words, using Porter's stemmer


[41]. After the pre-processing, each document was H4. The lexicon-based approach and the two Web-
represented as a keyword vector, which was used as the feature approaches will require significantly fewer
input to the SVM for training and testing. training data to achieve a satisfactory performance
than the keyword-based approach (SVM-WORD).
5.3. Implementation We suggest that the lexicon-based and the Web-
feature approaches require fewer training data because
All four approaches were implemented in order to they rely on only score(s) that should be similar across
test their performances. In the lexicon-based approach, Web pages. Only a small number of training samples
the Arizona Noun Phraser (AZNP) was used to extract would be needed for the classifiers to learn the
noun phrases from each document and these phrases importance of the scores. On the other hand, the
were compared with the domain lexicon. The AZNP is a traditional keyword-based approach needs the occur-
tool that extracts all valid noun phrases from a rence of certain keywords in order to classify a
document, based on part-of-speech tagging and linguis- document. When the number of training documents is
tic rules [53]. For the two approaches that rely on a small, it is likely that many words in the testing
support vector machine, the SVM-light package was documents have not been seen before and hence provide
used [24]. All other programs, including the feature no information for classification.
score calculation and the neural network algorithm,
were implemented in Java. 5.5. Experiment setup

5.4. Hypotheses Each of the four approaches was evaluated using
cross-validation, a widely-used evaluation methodology
Our experiment sought to compare the two Web- for machine learning and text classification systems
feature approaches with the two benchmark approaches. [27,43]. A 50-fold cross validation was adopted, in
We posed the following hypotheses: which the 1000 documents in the data set were divided
into 50 equal portions, with 20 documents each. Testing
was performed for 50 iterations, in each of which 49
H1. The keyword-based SVM approach (SVM-
portions of the data (980 documents) were used for
WORD) will perform with higher effectiveness than
training and the remaining portion (20 documents) was
the lexicon-based approach (LEXICON).
used for testing. The data were rotated during the
process such that each portion was used for testing in
We reasoned that a keyword-based approach should
exactly one iteration.
be able to make better classification decisions by relying
on more keyword information. We measured the effectiveness of each system using
precision, recall, F-measure, and accuracy. Precision
H2. The two proposed Web-feature approaches (NN- measures the fraction of the documents correctly
WEB and SVM-WEB) will perform with comparable classified as relevant, while recall measures the fraction
effectiveness. of relevant documents retrieved from the data set. F-
measure is a single measure that tries to combine
We hypothesized that the two proposed approaches precision and recall. Accuracy measures simply the
should perform similarly because both neural network prediction correctness of the classifiers. These measures
and support vector machine have been widely used in are commonly used in text classification evaluation and
text classification applications and should achieve have been adopted as follows:
comparable performance.
number of documents correctly classified as positive by the system
precision ¼
H3. The two proposed Web-feature approaches (NN- number of all documents classified as positive by the system

WEB and SVM-WEB) will perform with higher
number of documents correctly classified as positive by the system
effectiveness than the two benchmark approaches, i.e., recall ¼
number of positive documents in the testing set
the keyword-based SVM approach (SVM-WORD) and
the lexicon-based approach (LEXICON). 2 Â precision Â recall
FQmeasure ¼
precision þ recall
This hypothesis tests the main thesis of this paper by
verifying whether the proposed approaches perform number of documents correctly classified by the system
accuracy ¼
better than the traditional approaches. number of all documents in the testing set


There are two popular ways to calculate the averages Table 2
across the data for these metrics, namely, macro- Micro sign-test results
averaging and micro-averaging [4,18,23,32,51]. In vs. SVM-WORD NN-WEB SVM-WEB
macro-averaging, the performance metrics are calculated LEXICON b0.00001 b
b0.00001 b
b0.00001 b
for each iteration, and the average of all iterations is SVM-WORD 0.0972 a 0.3044
obtained. In micro-averaging, the average is calculated NN-WEB 0.0095 b
a
across all the individual classification decisions made by The difference is statistically significant at the 10% level.
b
a system. Both averages were calculated in our ex- The difference is statistically significant at the 1% level.
periment. In addition to effectiveness, we also recorded
the time used by each classifier in order to measure their
efficiencies. better than the lexicon-based approach with a p-value
less than 0.00001. H3 was partly supported; both the
5.6. Experiment results and discussions Web-feature approaches performed significantly better
than the lexicon-based approach, and the Web-feature
5.6.1. Effectiveness NN approach also performed significantly better than
The experiment results on accuracy, precision, recall, keyword-based SVM approach. H2 was not supported,
and F-measure are summarized in Table 1. Because F- since the Web-feature NN approach performed better
measures represent a balance between precision and than the Web-feature SVM approach.
recall, we focus our discussion on accuracy and F-mea- The p-values of the macro t-tests on accuracy and
sure. The results demonstrated that the lexicon-based F-measure are shown in Tables 3 and 4, respectively.
approach in general did not perform as well as the other The results obtained were similar to those of the micro
approaches; it achieved the lowest accuracy and F- sign-test. In general, H1 was supported as the keyword-
measure. NN-WEB achieved the highest accuracy and based approach performed significantly better than
F-measure. lexicon-based approach. H3 also was partly supported,
In order to study whether the differences among the since the Web-feature NN approach performed better
different approaches were statistically significant, two than both benchmark approaches, but the Web-feature
statistical tests were adopted. The first was a micro sign- SVM approach only performed better than the lexicon-
test that looks at all the classification decisions in- based approach but no better than the keyword-based
dividually and uses a binomial distribution to determine approach. H2 was not supported; the Web-feature NN
whether the decisions made by any two approaches of approach performed better than the Web-feature SVM
interest are significantly different [17]. The number of approach. One possible reason is that because of the
observations n is defined to be the number of times that limitation in resources and time, we were only able to use
the two systems made different classification decisions. the linear model in the SVM. It is possible that the
The second test was a macro t test that takes the per- performance of both SVM approaches might improve if
formance of each iteration as an individual observation a non-linear model could be adopted, which could be
in order to determine whether the performances of two more time-consuming, however.
approaches are significantly different [51]. Our number
of observations was 50, since there were 50 iterations of 5.6.2. Efficiency
testing for each approach. The macro t test was applied to We also recorded the time needed for each system to
both accuracy and F-measure. perform the 50-fold cross validation (including both
The p-values of the micro sign-tests are shown in training and testing time). The data are shown in
Table 2. The results show that hypothesis H1 was sup- Table 5. As can be seen, the keyword-based SVM
ported as the keyword-based SVM approach performed approach required the longest time. The reason is that

Table 1
Experiment results
Accuracy (%) Precision (macro/micro) (%) Recall (macro/micro) (%) F-measure (macro/micro)
LEXICON 80.80 63.40/63.95 60.52/62.50 0.6005/0.6322
SVM-WORD 87.80 87.97/94.94 55.08/56.82 0.6646/0.7109
NN-WEB 89.40 81.38/82.38 76.19/76.14 0.7614/0.7913
SVM-WEB 87.30 85.35/86.24 61.99/61.74 0.7049/0.7196


Table 3 Table 5
Macro t-test results on accuracy Time
vs. SVM-WORD NN-WEB SVM-WEB Time (min)
LEXICON b0.00001 b
b0.00001 b
b0.0001 b
LEXICON 7.45
SVM-WORD 0.1627 0.6091 SVM-WORD 382.55
NN-WEB 0.0216 a NN-WEB 103.45
a SVM-WEB 37.60
The difference is statistically significant at the 5% level.
b

Web-feature approaches became relatively stable after
each document was represented as a large vector of approximately 300, 140, and 260 training documents
keywords, which created a high dimensionality for the were used respectively. For the keyword-based ap-
classifier. In our experiment, there were more than 6000 proach, however, performance was unstable until about
unique words after stop-word removal and stemming. 700 training documents had been used. This supported
The classifier had to learn the relationships between all our hypothesis H4 that fewer documents were needed for
these attributes and the class attribute, thus requiring the lexicon-based approach or the Web-feature
more time. The lexicon-based approach used the least approaches to achieve a satisfactory performance. As
time, as it needed only to calculate the TFIDF and discussed earlier, this finding is especially important for
similarity scores for each document and determine the building vertical search engines as a large number of
threshold, both of which did not require complex pro- training documents often is not unavailable.
cessing. Comparing the two Web-feature approaches,
the NN classifier required a longer time than the SVM 6. Conclusion and future directions
classifier because the neural network had to be trained in
multiple epochs, i.e., in each iteration the training data In this paper, we have described a Web-feature ap-
set had to be presented to the network thousands of proach to Web page classification that combines Web
times in order to improve the network's performance. content analysis and Web structure analysis. We com-
pared our approaches with traditional text classification
5.6.3. Effect of the number of training examples methods and found the experimental results to be en-
In order to analyze the effect of the number of training couraging. We believe that the proposed approaches are
examples on the performance, we ran the experiments on useful for various Web applications, especially for ver-
the systems while varying the number of training data tical search engine development.
used. We started with 20 documents in the first run, and
increased the number of training documents by 20 in
each subsequent run. There were thus 49 runs in total
(from 20 to 980 training documents). In each run, a
50-fold cross validation similar to the one described
above was used, and 20 documents were used for testing
with rotation. The macro-averaged F-measure for each
iteration was recorded and the results are shown in
Fig. 1.
From the graph shown in Fig. 2, we can see that the
performances of the lexicon-based approach and the two

Table 4
Macro t-test results on F-measure
vs. SVM-WORD NN-WEB SVM-WEB
LEXICON 0.0827 a
b0.00001 b
0.0041 b
SVM-WORD 0.0024 b 0.2446
NN-WEB 0.0033 b
a
b
The difference is statistically significant at the 1% level. Fig. 2. F-measure vs. the number of training data.


While the Web-feature approaches are promising, it International ACM SIGIR Conference on Research and Devel-
is interesting to examine which of the 14 features used opment in Information Retrieval, Tampere, Finland, Aug 2002,
pp. 97–104.
are more important than the others in determining the [5] S. Chakrabarti, B. Dom, P. Indyk, Enhanced hypertext categori-
relevance of a page. We plan to apply factor analysis zation using hyperlink, Proceedings of ACM SIGMOD Interna-
techniques to the data set to investigate the features in tional Conference on Management of Data, Seattle, Washington,
detail. Another direction of our future work will be to USA, Jun 1998.
[6] S. Chakrabarti, B.E. Dom, S.R. Kumar, P. Raghavan, S.
study whether a combined keyword-based and Web-
Rajagopalan, A. Tomkins, D. Gibson, J. Kleinberg, Mining the
feature approach will perform better than using either web's link structure, IEEE Computer 32 (8) (1999) 60–67.
the keywords or the Web features alone. We believe that a [7] S. Chakrabarti, M. van den Berg, B. Dom, Focused crawling: a
combined approach may potentially acquire the strengths new approach to topic-specific web resource discovery, Proceed-
of both approaches and perform better by allowing the ings of the 8th International World Wide Web Conference,
classifier to rely on the feature scores when the number of Toronto, Canada, May 1999.
[8] M. Chau, H. Chen, Comparison of three vertical search spiders,
training documents is small but to rely more on unique IEEE Computer 36 (5) (2003a) 56–62.
keyword attributes in the vector when the number of [9] M. Chau, H. Chen, Personalized and focused Web spiders, in: N.
training documents reaches a certain level. Finally, we are Zhong, J. Liu, Y. Yao (Eds.), Web intelligence, Springer–Verlag,
also investigating how the proposed classification method 2003b, pp. 197–217.
can be used in other applications, such as knowledge [10] M. Chau, H. Chen, Incorporating Web analysis into neural
networks: an example in hopfield net searching, IEEE Transac-
management and Web content management. tions on Systems, Man, and Cybernetics (Part C) 37 (3) (2007)
352–358 (May).
Acknowledgements [11] M. Chau, Z. Huang, J. Qin, Y. Zhou, H. Chen, Building a
scientific knowledge web portal: the nanoport experience,
This project has been supported in part by the fol- Decision Support Systems 42 (2) (2006) 1216–1238.
[12] H. Chen, Machine learning for information retrieval: neural
lowing grants: networks, symbolic learning, and genetic algorithms, Journal of
the American Society for Information Science 46 (3) (1995)
• NSF Digital Library Initiative-2 (PI: H. Chen), 194–216.
“High-performance Digital Library Systems: From [13] H. Chen, Y. Chung, M. Ramsey, C.C. Yang, An intelligent
Information Retrieval to Knowledge Management,” personal spider (Agent) for dynamic Internet/Intranet searching,
Decision Support Systems 23 (1998) 41–58.
IIS-9817473, April 1999–March 2002; [14] H. Chen, A. Lally, B. Zhu, M. Chau, HelpfulMed: intelligent
• NIH/NLM Grant (PI: H. Chen), “UMLS Enhanced searching for medical information over the internet, Journal of
Dynamic Agents to Manage Medical Knowledge,” 1 the American Society for Information Science and Technology
R01 LM06919-1A1, February 2001–January 2004; 54 (7) (2003) 683–694.
[15] F.C. Cheong, Internet Agents: Spiders, Wanderers, Brokers, and
• HKU Seed Funding for Basic Research (PI: M. Chau),
Bots, New Riders Publishing, Indianapolis, Indiana, USA, 1996.
“Using Content and Link Analysis in Developing [16] J. Cho, H. Garcia-Molina, L. Page, Efficient crawling through
Domain-specific Web Search Engines: A Machine URL ordering, Proceedings of the 7th International World Wide
Learning Approach,” February 2004–July 2005. Web Conference, Brisbane, Australia, Apr 1998.
[17] W.W. Cohen, Text categorization and relational learning,
We would like to thank the National Library of Proceedings of the 12th International Conference on Machine
Learning (ICML'95), Morgan Kaufmann, 1995.
Medicine for making UMLS freely available to re- [18] W.W. Cohen, Y. Singer, Context-sensitive learning methods for
searchers, and the medical experts who participated in text categorization, ACM Transactions on Information Systems
the user studies. 17 (2) (1999) 141–173.
[19] M. Diligenti, F. Coetzee, S. Lawrence, C.L. Giles, M. Gori,
References Focused crawling using context graphs, Proceedings of the 26th
International Conference on Very Large Databases, VLDB 2000,
[1] A. Arasu, J. Cho, H. Garcia-Molina, A. Paepcke, S. Raghavan, Cairo, Egypt, 2000, pp. 527–534.
Searching the web, ACM Transactions on Internet Technology 1 [20] S.T. Dumais, J. Platt, D. Heckerman, M. Sahami, Inductive
(1) (2001) 2–43. learning algorithms and representations for text categorization,
[2] O. Baujard, V. Baujard, S. Aurel, C. Boyer, R.D. Appel, Trends Proceedings of ACM Conference on Information and Knowledge
in medical information retrieval on the Internet, Computers in Management, Bethesda, Maryland, Nov. 1998, pp. 148–155.
Biology and Medicine 28 (1998) 589–601. [21] J. Furnkranz, Exploiting structural information for text catego-
[3] S. Brin, L. Page, The anatomy of a large-scale hypertextual web rization on the WWW, Proceedings of the 3rd Symposium on
search engine, Proceedings of the 7th International World Wide Intelligent Data Analysis (IDA'99), Amsterdam, Netherlands,
Web Conference, Brisbane, Australia, Apr 1998. 1999, pp. 487–497.
[4] K.M.A. Chai, H.L. Chieu, H.T. Ng, Bayesian online classifiers for [22] M. Iwayama, T. Tokunaga, Cluster-based text categorization: a
text classification and filtering, Proceedings of the 25th Annual comparison of category search strategies, Proceedings of the


18th Annual International ACM Conference on Research [40] H.J. Oh, S.H. Myaeng, M.H. Lee, A practical hypertext
and Development in Information Retrieval (SIGIR'95), 1995, categorization method using links and incrementally available
pp. 273–281. class information, Proceedings of the 23rd Annual International
[23] T. Joachims, Text categorization with support vector machines: ACM Conference on Research and Development in Information
learning with many relevant features, Proceedings of the European Retrieval (SIGIR'00), 2000, pp. 264–271.
Conference on Machine Learning, Berlin, 1998, pp. 137–142. [41] M.F. Porter, An algorithm for suffix stripping, Program 14 (3)
[24] T. Joachims, Making large-Scale SVM Learning Practical, in: B. (1980) 130–137.
Schölkopf, C. Burges, A. Smola (Eds.), Advances in Kernel [42] E. Riloff, W. Lehnert, Information extraction as a basis for high-
Methods— Support Vector Learning, MIT-Press, 1999. precision text classification, ACM Transactions on Information
[25] T. Joachims, N. Chistianini, J. Shawe-Taylor, Composite kernels Systems 12 (3) (1994) 296–333.
for hypertext categorization, Proceedings of the 18th Interna- [43] M. Stone, Cross-validation choices and assessment of statistical
tional Conference on Machine Learning (ICML'01), 2001. predictions, Journal of the Royal Statistical Society 36 (1974)
[26] J. Kleinberg, Authoritative sources in a hyperlinked environ- 111–147.
ment, Proceedings of the ACM-SIAM Symposium on Discrete [44] A. Sun, E.-P. Lim, W.-K. Ng, Performance measurement
Algorithms, 1998. framework for hierarchical text classification, Journal of the
[27] R. Kohavi, A study of cross-validation and bootstrap for American Society for Information Science and Technology 54
accuracy estimation and model selection, Proceedings of the (11) (2003) 1014–1028.
14th International Joint Conference on Artificial Intelligence, [45] R.G. Sumner, K. Yang, B.J. Dempsey, An interactive www
San Francisco, CA, Morgan Kaufmann, 1995, pp. 1137–1143. search engine for user-defined collections, Proceedings of the 3rd
[28] D. Koller, M. Sahami, Hierarchically classifying documents using ACM Conference on Digital Libraries, Pittsburgh, Pennsylvania,
very few words, Proceedings of the 14th International Conference USA, Jun 1998, pp. 307–308.
on Machine Learning (ICML'97), 1997, pp. 170–178. [46] C.J. van Rijsbergen, Information Retrieval, Second Edition,
[29] R. Kumar, P. Raghavan, S. Rajagopalan, A. Tomkins, Trawling Butterworths, London, 1979.
the Web for emerging cyber-communities, Proceedings of the [47] V. Vapnik, The Nature of Statistical Learning Theory, Springer,
Eighth WWW Conference, Toronto, Canada, May 1999. 1995.
[30] Y. Labrou, T. Finin, Yahoo! as an ontology: using yahoo! categories [48] V. Vapnik, Statistical Learning Theory, Wiley, Chichester, GB,
to describe documents, Proceedings of the Eighth International 1998.
Conference on Information and Knowledge Management, Kansas [49] E. Wiener, J.O. Pedersen, A.S. Weigend, A neural network
City, Missouri, United States, 1999, pp. 180–187. approach to topic spotting, Proceedings of the 4th Annual
[31] S.L.Y. Lam, D.L. Lee, Feature reduction for neural network Symposium on Document Analysis and Information Retrieval
based text categorization, Proceedings of the International (SDAIR'95), 1995.
Conference on Database Systems for Advanced Applications [50] I.H. Witten, D. Bainbridge, S.J. Boddie, Greenstone: open-source
(DASFAA '99), Hsinchu, Taiwan, Apr 1999, 1999. DL software, Communications of the ACM 44 (5) (2001) 47.
[32] W. Lam, M. Ruiz, P. Srinivasan, Automatic text categorization [51] Y. Yang, X. Liu, A re-examination of text categorization
and its application to text retrieval, IEEE Transactions on methods, Proceedings of the 22nd Annual International ACM
Knowledge and Data Engineering 11 (6) (1999) 865–879. Conference on Research and Development in Information
[33] D.D. Lewis, M. Ringuette, Comparison of two learning algorithms Retrieval (SIGIR'99), 1999, pp. 42–49.
for text categorization, Proceedings of the Third Annual Sym- [52] Y. Yang, S. Slattery, R. Ghani, A study of approaches to hypertext
posium on Document Analysis and Information Retrieval categorization, Journal of Intelligent Information Systems 18 (2)
(SDAIR'94), 1994. (March 2002).
[34] S.H. Lin, M.C. Chen, J.M. Ho, Y.M. Huang, ACIRD: intelligent [53] K. Tolle, H. Chen, Comparing noun phrasing techniques for use
Internet document organization and retrieval, IEEE Transactions with medical digital library tools, Journal of the American
on Knowledge and Data Engineering 14 (3) (2002) 599–614. Society for Information Science 51 (4) (2000) 352–370.
[35] R.P. Lippmann, An introduction to computing with neural net-
works, IEEE Acoustics Speech and Signal Processing Magazine 4
(2) (1987) 4–22. Michael Chau is an Assistant Professor and
[36] U. Manber, M. Smith, B. Gopal, WebGlimpse: combining the BBA(IS)/BEng(CS) Coordinator in the
browsing and searching, Proceedings of the USENIX 1997 School of Business at the University of Hong
Annual Technical Conference, Anaheim, California, Jan 1997. Kong. He received his PhD degree in
[37] B. Masand, G. Linoff, D. Waltz, Classifying news stories using management information systems from the
memory based reasoning, Proceeedings of the 15th Annual University of Arizona and a bachelor degree
International ACM Conference on Research and Development in in computer science and information systems
Information Retrieval (SIGIR'92), 1992, pp. 59–64. from the University of Hong Kong. His
[38] A. McCallum, K. Nigam, J. Rennie, K. Seymore, A machine current research interests include informa-
learning approach to building domain-specific search engines, tion retrieval, Web mining, data mining,
Proceedings of the International Joint Conference on Artificial knowledge management, and security infor-
Intelligence (IJCAI-99), 1999, pp. 662–667. matics. He has published more than 60 research articles in leading
[39] H.T. Ng, W.B. Goh, K.L. Low, Feature selection, perceptron journals and conferences, including IEEE Computer, Journal of the
learning, and a usability case study for text categorization, America Society for Information Science and Technology, Decision
Proceedings of the 20th Annual International ACM Conference Support Systems, ACM Transactions on Information Systems, and
on Research and Development in Information Retrieval (SIGIR'97), Communications of the ACM. More information can be found at http://
1997, pp. 67–73. www.business.hku.hk/~mchau/.


Hsinchun Chen is a McClelland Professor as a Scientific Counselor/Advisor of the National Library of Medicine
of Management Information Systems at the (USA), Academia Sinica (Taiwan), and National Library of China
University of Arizona and Andersen Con- (China). He has been an advisor for major NSF, DOJ, NLM, DOD,
sulting Professor of the Year (1999). He DHS, and other international research programs in digital library,
received the B.S. degree from the National digital government, medical informatics, and national security
Chiao-Tung University in Taiwan, the MBA research. Dr. Chen is founding director of Artificial Intelligence Lab
degree from SUNY Buffalo, and the PhD and Hoffman E-Commerce Lab. He is conference co-chair of ACM/
degree in Information Systems from the IEEE Joint Conference on Digital Libraries (JCDL) 2004 and has
New York University. Dr. Chen is a Fellow served as the conference/program co-chair for the past eight
of IEEE and AAAS. He received the IEEE International Conferences of Asian Digital Libraries (ICADL), the
Computer Society 2006 Technical Achieve- premiere digital library meeting in Asia that he helped develop. Dr.
ment Award. He is author/editor of 13 books, 17 book chapters, and Chen is also (founding) conference co-chair of the IEEE International
more than 130 SCI journal articles covering intelligence analysis, Conferences on Intelligence and Security Informatics (ISI) 2003–
biomedical informatics, data/text/web mining, digital library, knowl- 2007. Dr. Chen has also received numerous awards in information
edge management, and Web computing. Dr. Chen was ranked #8 in technology and knowledge management education and research
publication productivity in Information Systems (CAIS 2005) and #1 including: ATT Foundation Award, SAP Award, the Andersen
in Digital Library research (IPM 2005) in two recent bibliometric Consulting Professor of the Year Award, the University of Arizona
studies. He serves on ten editorial boards including: ACM Transactions Technology Innovation Award, and the National Chaio-Tung Uni-
on Information Systems, IEEE Transactions on Systems, Man, and versity Distinguished Alumnus Award. Further information can be
Cybernetics, Journal of the American Society for Information Science found at http://ai.arizona.edu/hchen/.
and Technology, and Decision Support Systems. Dr. Chen has served

A machine learning approach to web page filtering using ...

More Related Content

What's hot (19)

Viewers also liked (8)

Similar to A machine learning approach to web page filtering using ... (20)

More from butest (20)

A machine learning approach to web page filtering using ...