Webmining Overview
0 likes•438 views
Web mining involves analyzing textual and link structure data from the world wide web to discover useful information. It deals with petabytes of data generated daily and needs to adapt to evolving usage patterns in real-time. Topics related to web mining include web graph analysis, power laws, structured data extraction, web advertising, user analysis, social networks, and blog analysis. The future will involve very large-scale data mining of datasets too big to fit in memory or even on a single disk.
Webmining Overview
- 1. Introduction to Web Mining
- 2. What is Web Mining? Discovering useful information from the World Wide Web ( such as web pages, internet related data and so forth) Example of applications: User patterns analysis Web page link analysis And more
- 3. Web Mining Involves Textual information and linkage structure analysis Peta bytes of data generated per day is comparable to largest conventional data warehouses in world Often need to react to evolving usage patterns in real-time (e.g., merchandising) and also accommodate the changes.
- 4. Topics related to web mining Web graph analysis Power Laws and The Long Tail Structured data extraction Web advertising Systems Issues, User analysis Social network analysis, blog analysis
- 5. Size of the Web Number of pages Technically, infinite Much duplication (30-40%) Growing everyday Best estimate of “unique” static HTML pages comes from search engine claims Google recently announced that their index contains 1 trillion pages
- 6. The web as a graph Pages = nodes, hyperlinks = edges Ignore content Directed graph High linkage 10-20 links/page on average Power-law degree distribution
- 7. Web graph Let’s take a closer look at structure Broder et al (2000) studied a crawl of 200M pages and other smaller crawls Distinguish “important” pages from unimportant ones Page rank Discover communities of related pages Hubs and Authorities Detect web spam Trust rank
- 8. Searching the Web Content consumers Content aggregators The Web
- 9. Two Approaches to Analyzing Data Machine Learning approach Emphasizes sophisticated algorithms e.g., Support Vector Machines Data sets tend to be small, fit in memory Data Mining approach Emphasizes big data sets (e.g., in the terabytes) Data cannot even fit on a single disk! Necessarily leads to simpler algorithms
- 10. The future: Very Large-Scale Data Mining … Mem Disk CPU Mem Disk CPU Mem Disk CPU
- 11. Visit more self help tutorials Pick a tutorial of your choice and browse through it at your own pace. The tutorials section is free, self-guiding and will not involve any additional support. Visit us at www.dataminingtools.net