Hadoop - Introduction to Hadoop

HadoopHadoop IntroductionIntroduction

Data Scalability ProblemsData Scalability Problems
• Search Engine
o 10KB / doc * 20B docs = 200TB
o Reindex every 30 days: 200TB/30days = 6 TB/day
• Log Processing / Data Warehousing
o 0.5KB/events * 3B pageview events/day = 1.5TB/day
o 100M users * 5 events * 100 feed/event * 0.1KB/feed = 5TB/day
• Multipliers: 3 copies of data, 3-10 passes of raw data
• Processing Speed (Single Machine)
o 2-20MB/second * 100K seconds/day = 0.2-2 TB/day

Google’s SolutionGoogle’s Solution
• Google File System – SOSP’2003
• Map-Reduce – OSDI’2004
• Sawzall – Scientific Programming Journal’2005
• Big Table – OSDI’2006
• Chubby – OSDI’2006

Open Source World’s SolutionOpen Source World’s Solution
• Google File System – Hadoop Distributed FS
• Map-Reduce – Hadoop Map-Reduce
• Sawzall – Pig, Hive, JAQL
• Big Table – Hadoop HBase, Cassandra
• Chubby – Zookeeper

Hadoop HistoryHadoop History
• Jan 2006 – Doug Cutting joins Yahoo
• Feb 2006 – Hadoop splits out of Nutch and Yahoo starts using it.
• Dec 2006 – Yahoo creating 100-node Webmap with Hadoop
• Apr 2007 – Yahoo on 1000-node cluster
• Jan 2008 – Hadoop made a top-level Apache project
• Dec 2007 – Yahoo creating 1000-node Webmap with Hadoop
• Sep 2008 – Hive added to Hadoop as a contrib
project

Hadoop IntroductionHadoop Introduction
• Written in Java
o Does work with other languages
• Runs on
o Linux, Windows and more
o Commodity hardware with high failure rate

Current Status of HadoopCurrent Status of Hadoop
• Largest Cluster
o 2000 nodes (8 cores, 4TB disk)
• Used by 40+ companies / universities over the world
o Yahoo, Facebook, etc
o Cloud Computing Donation from Google and IBM
• Startup focusing on providing services for hadoop
o Cloudera

Hadoop ComponentsHadoop Components
• Hadoop Distributed File System (HDFS)
• Hadoop Map-Reduce
• Contributes
o Hadoop Streaming
o Pig / JAQL / Hive
o HBase
o Hama / Mahout

Hadoop Distributed File
System

Goals ofGoals of HDFSHDFS
• Very Large Distributed File System
o 10K nodes, 100 million files, 10 PB
• Convenient Cluster Management
o Load balancing
o Node failures
o Cluster expansion
• Optimized for Batch Processing
o Allow move computation to data
o Maximize throughput

HDFS ArchitectureHDFS Architecture

HDFS DetailsHDFS Details
• Data Coherency
o Write-once-read-many access model
o Client can only append to existing files
• Files are broken up into blocks
o Typically 128 MB block size
o Each block replicated on multiple DataNodes
• Intelligent Client
o Client can find location of blocks
o Client accesses data directly from DataNode

HDFS User InterfaceHDFS User Interface
• Java API
• Command Line
o hadoop dfs -mkdir /foodir
o hadoop dfs -cat /foodir/myfile.txt
o hadoop dfs -rm /foodir myfile.txt
o hadoop dfsadmin -report
o hadoop dfsadmin -decommission datanodename

More about HDFSMore about HDFS
•Hadoop FileSystem API
o HDFS
o Local File System
o Kosmos File System (KFS)
o Amazon S3 File System

Hadoop Map-Reduce and
Hadoop Streaming

Hadoop StreamingHadoop Streaming
• Allow to write Map and Reduce functions in any
languages
o Hadoop Map/Reduce only accepts Java
• Example: Word Count
o hadoop streaming
-input /user/zshao/articles
-mapper ‘tr “ ” “n”’
-reducer ‘uniq -c‘
-output /user/zshao/
-numReduceTasks 32

Example: Log ProcessingExample: Log Processing
• Generate #pageview and #distinct users
for each page each day
o Input: timestamp url userid
• Generate the number of page views
o Map: emit < <date(timestamp), url>, 1>
o Reduce: add up the values for each row
• Generate the number of distinct users
o Map: emit < <date(timestamp), url, userid>, 1>
o Reduce: For the set of rows with the same <date(timestamp), url>, count the
number of distinct users by “uniq –c"

Example: PageExample: Page RankRank
• In each Map/Reduce Job:
o Map: emit <link, eigenvalue(url)/#links>
for each input: <url, <eigenvalue, vector<link>> >
o Reduce: add all values up for each link, to generate the new eigenvalue for
that link.
• Run 50 map/reduce jobs till the eigenvalues are
stable.

TODOTODO:: Split Job Scheduler and Map-ReduceSplit Job Scheduler and Map-Reduce
• Allow easy plug-in of different scheduling algorithms
o Scheduling based on job priority, size, etc
o Scheduling for CPU, disk, memory, network bandwidth
o Preemptive scheduling
• Allow to run MPI or other jobs on the same cluster
o PageRank is best done with MPI

Map-Reduce and SQLMap-Reduce and SQL
• Map-Reduce is scalable
o SQL has a huge user base
o SQL is easy to code
• Solution: Combine SQL and Map-Reduce
o Hive on top of Hadoop (open source)
o Aster Data (proprietary)
o Green Plum (proprietary)

HiveHive
• A database/data warehouse on top of Hadoop
o Rich data types (structs, lists and maps)
o Efficient implementations of SQL filters, joins and group-by’s on
top of map reduce
• Allow users to access Hive data without using Hive

Dealing with Structured DataDealing with Structured Data
• Type system
o Primitive types
o Recursively build up using Composition/Maps/Lists
• Generic (De)Serialization Interface (SerDe)
o To recursively list schema
o To recursively access fields within a row object
• Serialization families implement interface
o Thrift DDL based SerDe
o Delimited text based SerDe
o You can write your own SerDe
• Schema Evolution

MetaStoreMetaStore
• Stores Table/Partition properties:
o Table schema and SerDe library
o Table Location on HDFS
o Logical Partitioning keys and types
o Other information
• Thrift API
o Current clients in Php (Web Interface), Python (old CLI), Java (Query Engine and
CLI), Perl (Tests)
• Metadata can be stored as text files or even in a SQL
backend

Hive CLIHive CLI
• DDL:
o create table/drop table/rename table
o alter table add column
• Browsing:
o show tables
o describe table
o cat table
• Loading Data
• Queries

Web UI for HiveWeb UI for Hive
• MetaStore UI:
o Browse and navigate all tables in the system
o Comment on each table and each column
o Also captures data dependencies
• HiPal:
o Interactively construct SQL queries by mouse clicks
o Support projection, filtering, group by and joining
o Also support

Hive Query LanguageHive Query Language
• Philosophy
o SQL
o Map-Reduce with custom scripts (hadoop streaming)
• Query Operators
o Projections
o Equi-joins
o Group by
o Sampling
o Order By

Hive QL – Custom Map/Reduce ScriptsHive QL – Custom Map/Reduce Scripts
• Extended SQL:
• FROM (
• FROM pv_users
• MAP pv_users.userid, pv_users.date
• USING 'map_script' AS (dt, uid)
• CLUSTER BY dt) map
• INSERT INTO TABLE pv_users_reduced
• REDUCE map.dt, map.uid
• USING 'reduce_script' AS (date, count);
• Map-Reduce: similar to hadoop streaming

ThankThank You !!!You !!!
For More Information click below link:
Follow Us on:
http://vibranttechnologies.co.in/hadoop-classes-in-mumbai.html

Hadoop - Introduction to Hadoop

More Related Content

What's hot (20)

Similar to Hadoop - Introduction to Hadoop (20)

More from Vibrant Technologies & Computers (20)

Recently uploaded (20)

Hadoop - Introduction to Hadoop

Editor's Notes