Download to read offline
Web Mining.pptx
- 1. Chapter-6 Web mining Data miningin web applications
- 2. What isWeb mining? Webmining is the use of data mining techniques to automatically discover and extract information fromWeb documents and services.
- 3. What is webmining? • Mining of data related toWWW • Data present inWeb pages or data related to web activity • Web data is classified • Content of web pages • Intra page structure which include code and actual linkage • Usage data – how used by visitors • User profiles
- 4.
- 5. Web Content Mining •Extension of basic search engines • Search engines are keyword-based • Traditional search engines use crawlers • to search theWeb • gather information • indexing techniques to store the information • query processing to provide fast and accurate information to users
- 6. Taxonomy ofWeb contentmining AGENT BASED APPROACH WEB CONTENT MINING DATABASEAPPROACH USE SOFTWARE SYSTEMSTO PERFORM THE CONTENT MINING EG. SEARCH ENGINES VIEWSWEB DATA AS BELONGINGTO DATABASE WEB IS A MULTILEVEL DATABASEAND QUERY LANGUAGESARE USED FOR QUERYINGTHE DATA CONTENT MINING ISATYPE OF TEXT MINING
- 7. Text mining hierarchy Keyword Term Association SimilaritySearch Classification and Clustering Natural Language processing Simple Complex
- 8.
- 9. How do crawlerswork? • Robot, spider, crawler is a program that traverses the hypertext structure in the web • Page that the crawler starts is referred to as seed URL • All links from that page are recorded and saved in a queue • The new pages are in turn searched and their links are saved • The crawlers collect information about each page, extract keywords, store indices for users
- 10. Crawling the web AWeb crawler is an Internet bot which systematically browses the World Wide Web, typically for the purpose of Web indexing (web spidering). Web search engines and some other sites use Web crawling or spidering software to update their web content or indices of others sites' web content. Including a robots.txt file can request bots to index only parts of a website, or nothing at all.
- 11. Crawling When the Googlevisit your website for the purpose of tracking, Google does this with help of machine, known as web crawler, spider, Google bot, internet bot, automatic indexer The process of Crawling: Google uses huge set of computer to fetch or crawl” million of web pages on the web. Googlebot discovers new and updated pages with the help of site map to be added to the Google to crawl. Crawler obtains information; add in to the Google index, this is where crawling works.
- 12. Indexing Once the crawlingprocess has been done from the web crawler, the result will store in the Google index. The Google index is alike to an index or library, which lists information about all the books or library. If you want more pages included in the Google index, you can create and submit a Sitemap through Webmaster Tools. The index is basically a big list of words and the web pages that feature them on the basis of keywords location of the term in that particular webpage will store.
- 13. Ranking “Ranking – Determiningwhat each page is about, and how it should rank for relevant queries” Search engines have two major functions: crawling and building an index, and providing search users with a ranked list of the websites they've determined as the most relevant.
- 14. Types of crawlers •Periodic crawlers: activated periodically; every time it is activated it replaces the existing index • Incremental crawler: updates the index incrementally instead of replacing it • Focused crawler: visits pages related to topics of interest
- 15. Focused crawling SEED URL SITE-2SITE-3 SITE-1 SITE-6 SITE-5 SITE-4
- 16. Web Harvesting • Webharvesting also known asWeb Scraping is in an increasingly popular method used by websites to channel customer’s searches to their website • Web harvesting software automatically extracts information from the Web and picks up where search engines leave off, doing the work the search engine can't. • FMiner
- 17. VirtualWebView • Large amountof unstructured data can be handled using multiple layered database(MLDB) on top of the web data • Every layer of this dbase is more generalized then the preceding layer • The upper layer are structured and can be accessed using SQL • View of MLDB-Virtual WebView(VWV)
- 18. WebML • Query languagewhich supports data mining operations on MLDB • Four primitive operations inWebML are • COVERS • COVERED BY • LIKE • CLOSETO SELECT * FROM document in “www.engr.smu.edu” WHERE ONE OF keywords COVERS “cat”
- 19. Personalization • Contents ofa web page are modified to fit the desires of the user • Advertisements are sent to a potential customer based on his specific knowledge • Personalization is performed on target web page • Targeting is different from personalization • In targeting businesses display advertisements at other sites visited by their users • In personalization when a person visits aWeb site, the advertising can be designed specifically for that person
- 20. Personalization Contd…. • Personalizationis a combination of clustering, classification and prediction • Types of personalization are • Manual techniques – user registration details • Collaborative filtering • Content-based filtering • Eg. MyYahoo
- 21. Web Usage Mining Itdeals with understanding user behavior in interacting with the web or with a website. Aim To obtain information that may assist web sites for reorganization or adaptation to better suit the user.
- 22. •Clicking pattern •Browsing time •Transaction Tounderstand user’s behaviour
- 23. Web Usage MiningApplications • Personalization • Improve structure of a site’s Web pages • Aid in caching and prediction of future page references • Improve design of individual pages • Improve effectiveness of e-commerce (sales and advertising)
- 24. Web Usage MiningActivities • Preprocessing Web log • Cleanse • Remove extraneous information • Sessionize Session: Sequence of pages referenced by one user at a sitting. • Pattern Discovery • Count patterns that occur in sessions • Pattern is sequence of pages references in session. • Similar to association rules • Transaction: session • Itemset: pattern (or subset) • Order is important • Pattern Analysis
- 25. Web Usage MiningIssues • Identification of exact user not possible. • Exact sequence of pages referenced by a user not possible due to caching. • Session not well defined • Security, privacy, and legal issues
- 26. Web Log Cleansing •Replace source IP address with unique but non-identifying ID. • Replace exact URL of pages referenced with unique but non-identifying ID. • Delete error records and records containing not page data (such as figures and code)
- 27. Web Structure Mining •Creating a model of the web organization • Used to classifyWeb pages or to create similarity measures between documents • Mine structure (links, graph) of theWeb • Techniques • PageRank • HITS
- 28. Page Rank • Designedto increase the effectiveness of search engines and improve their efficiency • Used to • Measure the importance of a page • Prioritize the pages returned from a traditional search engine using keyword searching • Page Rank is calculated based on the number of pages that point to it
- 29. Page Rank Searchengine that uses link structure to calculate a quality ranking (PageRank) for each page Intuition: PageRank can be seen as the probability that a “random surfer” visits a page A page is important if important pages link to it
- 30. PageRank Page Rank: Apage is important if many important pages link to it. (PageRank) + (Website Content) = Overall Rank in Results Link ij : i considers j important. the more important i, the more important j becomes. if i has many out-links: links are less important.
- 31. Let OutDegreei =# out-links of page i Adjust pj: PageRank ( j ) (1 d ) +d PageRank (i ) OutDegree(i) This is the weighted sum of the importance of the pages referring to Pj d-damping factor Parameter d is probability that the surfer gets bored and starts on a new random page (1-d) is the probability that the random surfer follows a link on current page
- 32. Repeat until pagerankvector converges…
- 33. Hyperlink-induced topic search(HITS) •Finds hubs and authoritative pages • HITS has two components • Based on a given set of keywords relevant pages are found • Hubs and authority measures are associated with these pages. Pages with highest values are returned
- 34. Authorities and hubs •The algorithm produces two types of pages: - Authority: pages that provide an important, trustworthy information on a given topic (highly-referenced pages on a topic) - Hub: pages that “point” to authorities • A better hub points to many good authorities .A better authority is pointed to by many good hubs
- 35. Definitions • Authority: pagesthat provide an important, trustworthy information on a given topic • Hubs: pages that contain links to authorities • Indegree: number of incoming links to a given node, used to measure the authoritativeness • Outdegree: number of outgoing links from a given node, here it is used to measure the hubness
- 36. 36 HITS Algorithm • Hubspoint to lots of authorities. • Authorities are pointed to by lots of hubs. • Together they form a bipartite graph: Hubs Authorities
- 37. HITS Pages that linkto a collection of authoritative pages on a broad topic Hubs
- 38. Authorities Relevant pages ofthe highest quality on a broad topic
- 39. HITS Steps forDiscovering Hubs and Authorities on a specific topic Collect seed set of pages S (returned by search engine) Expand seed set to contain pages that point to or are pointed to by pages in seed set (removes links inside a site) Iteratively update hub weight h(p) and authority weight a(p) for each page: a (p ) h(q ) h(p ) a (q ) q p p q After a fixed number of iterations, pages with highest hub/authority weights form core of community
- 40. Strengths and weaknessesof HITS Strength: its ability to rank pages according to the query topic, which may be able to provide more relevant authority and hub pages. Weaknesses: It is easily spammed. It is in fact quite easy to influence HITS since adding out-links in one’s own page is so easy. Topic drift. Many pages in the expanded set may not be on topic. Inefficiency at query time: The query time evaluation is slow.
- 41.
- 42. Application areas ofweb mining 1. E-commerce: personalized marketing; 2. Fight against terrorism: classify threats; 3. Prediction; 4. And others :)
- 43. Future research directions 1.Multimedia data mining: a picture is worth a thousand words; 2. Multilingual knowledge extraction: web page translations; 3. Semantic web mining.
- 44.