Web Mining

Presented By: Akshat Saxena Anjul Sahu

Definition Application of data mining techniques on the web to discover interesting patterns.

Introduction Size of web is extremely large Data present on web is unstructured Good scope of data mining Types of data on web Content of actual webpage Intrapage structure Interpage structure Usage data User profiles and cookies

Web Content Mining Extends work of search engine Improves on traditional crawler technique Use data mining for efficiency, effectiveness and scalability Further divided into Agent based approach Database based approach Text mining is/isn’t content mining Crawlers Personalization

Web Content Mining Subtasks Resource finding Retrieving intended documents Information selection/pre-processing Select and pre-process specific information from selected documents Generalization Discover general patterns within and across web sites Analysis Validation and/or interpretation of mined patterns

Web Crawler Program which browses WWW in a methodical, automated manner Copy in cache and do Indexing Starts from a seed url Searches and finds links, keywords Types of Crawler Context focused Focused Incremental Periodic

Focused Crawler Visits only pages of interest Architecture consists of: Hyperlink Classifier Distiller Crawler Hub pages - links to relevant pages Hard focus - parent node relevant Soft focus - probability of relevance Harvest rate – precision rate

Context Focused Crawler Focused crawler was static Drawbacks: Non-relevant pages having links to relevant ones. These to be followed Relevant ones not having links to other relevant ones. Backward crawling CFC in two steps Construct context graphs and classifiers Crawl using these classifiers

Harvest System Uses caching, indexing and crawling Act as a tool in gathering information from other sources Components: Gatherer - obtains information Broker - provides index and query interface Essence systems Semantic indexing

Virtual Web View Web as multiple layer database A view of MLDB is virtual web view No spiders used Websites send their indices to others WebML – DMQL for web mining KEYWORDS – covers, covered by, like, close to Difficult to implement

Personalization Contents of web are modified as per user’s desires Personalized not targeted Use cookies, userID, profile information Legal issues to be considered Includes clustering, classification or even prediction

Personalization Types: User preference Collaborative filtering Content based filtering Example : My Yahoo! was first. Now almost every service offers personalization.

Personalization Yahoo was the first to introduce the concept of a ’personalized portal’, i.e. a Web site designed to have the look-and-feel as well as content personalized to the needs of an individual end-user. Mining MyYahoo usage logs provides Yahoo valuable insight into an individual’s Web usage habits, enabling Yahoo to provide compelling personalized content, which in turn has led to the tremendous popularity of the Yahoo Web site.

Web Structure Mining Creating a model of web organization Classify web pages Create similarity measures between web pages Page Rank The Clever system Hyperlink induced topic search(HITS)

PageRank TM Link analysis algorithm which assigns numerical weight to a webpage. The numerical weight that it assigns to any given element E is also called the PageRank of E and denoted by PR(E). the PageRank value for a page u is dependent on the PageRank values for each page v out of the set B u (this set contains all pages linking to page u ), divided by the number L ( v ) of links from page v .

Page Rank Increase effectiveness of search engines Based on number of back links Rank sink problem exists

Clever System Finds both authoritative pages and hubs Authoritative - best source Hub - link to authoritative pages Most value page returned Hyperlink Induced Topic Search Keywords Authority and hub measure

Alternatives to PageRank HITS Algorithm IBM Clever Project TrustRank But PageRank is the most popular and widely used algorithm by search engines

Web Usage Mining Applies mining on web usage data or weblogs or clickstream data Client perspective Server perspective Aid in personalization Helps in evaluating quality and effectiveness Preprocessing, pattern discovery and data structures

Trackers for site usage and analysis

Issues in Web Log Identify exact user Exact sequence of pages visited Security, privacy and legal issues

Preprocessing Information not in presentable format Data cleaning required Log: (<src id>,<literal>,<timestamp>) Data might be grouped Sessions Path completion

Data Structure DS needed to keep track of patterns identified DS used is trie A rooted tree where each path from root to node represents a sequence

Pattern Discovery Traversal pattern - pages visited in a session Properties: Duplicate reference may / may not be allowed Consist of only contiguous page reference Pattern may / may not be maximal Association rules - pages accessed together

Pattern Discovery Sequential Pattern - ordered set satisfying a support and maximal Similar to apriori algorithm Web access pattern - efficient counting Episodes – partially ordered by access time; users not identified Pattern analysis

Queries ‘N Suggestions References: http://maya.cs.depaul.edu/~mobasher/webminer/survey/ Google.com/Technology http://www.almaden.ibm.com/projects/clever.shtml Thanks !!  {akshatsaxena11, anjulsahu}@gmail.com

Web Mining

More Related Content

What's hot (20)

Similar to Web Mining (20)

Recently uploaded (20)

Web Mining