BIg Data Analytics-Module-2 vtu engineering.pptx

Module-2
Introduction to Hadoop

LEARNING OBJECTIVES
- LO 2.1 Get conceptual understanding of Hadoop core, components of Hadoop
ecosystem, and streaming and pipe interfaces for inputs to MapReduce
- LO 2.2 Get understanding of Hadoop Distributed File System (HDFS), and
physical-organization of nodes for computing at clusters of large-scale files
- LO 2.3 Get knowledge of MapReduce Daemon framework, and MapReduce
programming model
- LO 2.4 Get knowledge of functions of Hadoop YARN, management and
scheduling of resources, and parallel processing of the application-tasks
- LO 2.5 Get introduced to functions of Hadoop ecosystem-tools

RECALL
Requirements for Big Data processing and analytics are:
- 1. Huge volume of data stores
- 2. Flexible, scalable and distributed storage and computations
- 3. Distributed data blocks and tasks, and processing at the clusters
- 4. Mapping of the data at the physical nodes
- 5. Reducing the complexity of multiple data sources, and sending the computational results to
the applications
- 6. Developing, deploying, operating and managing applications in Big Data environment.
- 7. Integration of solutions into a cohesive solution
- 8. Uses of large resources of MPPs, cloud, specialized tools and parallel processing and use of
high speed networks

Here we focuses on
- Hadoop,
- its ecosystem,
- HDFS based programming model,
- MapReduce,
- Yarn, and
- introduces to ecosystem components,
such as AVRO, Zookeeper, Ambari, HBase, Hive, Pig and Mahout.

2.1 INTRODUCTION
Centralized Computing Model:
- A programming model is centralized computing of data in which the data is
transferred from multiple distributed data sources to a central server.
- Analyzing, reporting, visualizing, business-intelligence tasks compute centrally.
- Centralized server does the function of collection, storing and analyzing.
- Example, at an ACVM Company enterprise server. The data at the server gets
collected from a large number of ACVMs which the company locates in multiple
cities, areas and locations.
- The server also receives data from social media.

Applications running at the server does the following analysis:
1. Suggests a strategy for filling the machines at minimum cost of logistics
2. Finds locations of high sales such as gardens, playgrounds etc.
3. Finds days or periods of high sales such as Xmas etc.
4. Finds children's preferences for specific chocolate flavors
5. Finds the potential region of future growth
6. Identifies unprofitable machines
7. Identifies need of replicating the number of machines at specific
locations.

Distributed Computing Model:
- Another programming model is distributed computing model that uses the
databases at multiple computing nodes with data sharing between the nodes
during computation.
- Distributed computing in this model requires the cooperation (sharing)
between the DBs in a transparent manner.
- Transparent means that each user within the system may access all the data
within all databases as if they were a single database.
- A second requirement is location independence. Analysis results should be
independent of geographical locations.

Example shows why the simply scaling out and division of the computations on a
large number of processors may not work well due to data sharing between
distributed computing nodes.
Consider a jigsaw Puzzle (5000 pieces). Children above 14 years of age will
assemble the pieces in order to solve the puzzle. What will be the effect on time
intervals for solution in three situations, when 4, 100 and 200 children
simultaneously attempt the solution.

Transparency between data nodes at computing nodes do not fulfil all the needs of Big
Data when distributed computing takes place using data sharing between local and
remote.
Following are the reasons for this:
• Distributed data storage systems do not use the concept of joins.
• Data need to be fault-tolerant and data stores should take into account the
possibilities of network failure. When data need to be partitioned into data blocks
and written at one set of nodes, then those blocks need replication at multiple
nodes. This takes care of possibilities of network faults. When a network fault
occurs, then replicated node makes the data available.
• Big Data follows a theorem known as the CAP theorem. The CAP states that out of
three properties (consistency, availability and partitions), two must at least be
present for applications, services and processes.

(i) Big Data Store Model:
• The data blocks are distributed across multiple nodes.
• Data nodes are at the racks of a cluster.
• Racks are scalable.
• A Rack has multiple data nodes (data servers), and each cluster is arranged in
a number of racks.
• Data blocks replicate at the Data Nodes such that a failure of link leads to
access of the data block from the other nodes replicated at the same or other
racks.

(ii) Big Data Programming Model:
• Big Data programming model is an application in which jobs and tasks (or
sub-tasks) is scheduled on the same servers which store the data for processing.
• Job means running an assignment of a set of instructions for processing.
• For example, processing the queries in an application and sending the result back
to the application is a job.
• Job scheduling means assigning a job for processing following a schedule.
• For example, scheduling after a processing unit finishes the previously assigned
job, scheduling as per specific sequence or after a specific period.

• Hadoop system uses this programming model, where jobs or tasks are
assigned and scheduled on the same servers which hold the data.
• Hadoop is one of the widely used technologies.
• Google and Yahoo use Hadoop. Hadoop creators created a cost-effective
method to build search indexes.
• Facebook, Twitter and Linkedln use Hadoop.
• IBM implemented Biglnsights application using licensed Apache Hadoop.

- Following are important key terms and their meaning.
Cluster Computing:
• Refers to storing, computing and analyzing huge amounts of unstructured or
structured data in a distributed computing environment.
• Each cluster is formed by a set of loosely or tightly connected computing
nodes that work together and many of the operations can be timed and can be
realized as if from a single computing system.
• Clusters improve the performance, provide cost-effective and improved
node accessibility compared to a single computing node.

- Data Flow (DF)
refers to flow of data from one node to another.
For example, transfer of output data after processing to input of application.
-Data Consistency
means all copies of data blocks have the same values.
- Data Availability
• means at least one copy is available in case a partition becomes inactive or fails.
• For example, in web applications, a copy in the other partition is available. Partition
means parts, which are active but may not cooperate as in distributed databases
(DBs).

- Resources
means computing system resources, i.e., the physical or virtual components
or devices, made available for specified or scheduled periods within the system.
Resources refer to sources such as files, network connections and memory
blocks.
- Resource management
Refers to managing resources such as their creation, deletion and controlled
usages. The manager functions also includes managing the
(i) availability for specified or scheduled periods,
(ii)resources allocation when multiple tasks attempt to use the same set of
resources.

Horizontal scalability
Means increasing the number of systems working in coherence.
For example, using MPPs or number of servers as per the size of the dataset.
Vertical scalability
Means scaling up using the giving system resources and increasing the number of
tasks in the system.
For example, extending analytics processing by including the reporting, business
processing (BP), business intelligence (BI), data visualization, knowledge
discovery and machine learning (ML) capabilities require the use of
high capability hardware resources.

Ecosystem
Refers to a system made up of multiple computing components, which
work together. That is similar to a biological ecosystem, a complex
system of living organisms
Distributed File System
• Means a system of storing files. Files can be for the set of data records,
key- value pairs, hash key-value pairs, relational database or
NoSQL database at the distributed computing nodes.
• Data is accessed from distributed file system using a master directory
service, look-up tables or name-node server.

Hadoop Distributed File System
means a system of storing files (set of data records, key-value pairs, hash
key-value pairs or applications data) at distributed computing nodes.
Uses NameNode servers to enable finding data blocks in racks and cluster.

Scalability of storage and processing
Means the execution using varying number of servers according to the
requirements,
i.e., bigger data store on greater number of servers when required and on
smaller data when smaller data used on limited number of servers.
Big Data Analytics require deploying the clusters using the servers or cloud
for computing as per the requirements.

Utility Cloud-based Services
Means infrastructure, software and computing platform services similar
to utility services, such as electricity, gas, water etc.
Infrastructure refers to units for data-store, processing and network. The
IaaS, SaaS and PaaS are the services at the cloud.

2.21 HADOOP AND ITS ECOSYSTEM
Apache initiated the project for developing storage and processing framework for Big Data
storage and processing.
Doug Cutting and Machael j. the creators named that framework as Hadoop.
Cutting's son was fascinated by a toy elephant, named Hadoop, and this is how the name
Hadoop was derived.
The project consisted of two components,
- one of them is for data store in blocks in the clusters and
- the other is computations at each individual cluster in parallel with another.
Hadoop components are written in Java with part of native code in C.
The command line utilities are written in shell scripts.

• Hadoop Infrastructure consists of cloud for clusters.
• A cluster consists of sets of computers or PCs.
• The Hadoop platform provides a low cost Big Data platform, which is open
source and uses cloud services.
• Tera Bytes of data processing takes just few minutes.
• Hadoop characteristics are scalable, self-manageable, self-healing and
distributed file system.

Scalable means can be scaled up (enhanced) by adding storage and processing
units as per the requirements.
Self-manageable means creation of storage and processing resources which are
used, scheduled and reduced or increased with the help of the system itself.
Self-healing means that in case of faults, they are taken care of by the system
itself. Self-healing enables functioning and resources availability.
Software detect and handle failures at the task level.
Software enable the service or task execution even in case of communication or
node failure.

• The hardware scales out from a single server to thousands of machines that
store the clusters.
• Each cluster stores a large number of data blocks in racks. Default data block
size is 64 MB.
• IBM Biglnsights application, built on Hadoop deploys default 128 MB block size.

• Hadoop enables Big Data storage and cluster computing.
• The Hadoop system manages both, large-sized structured and unstructured data
in different formats, such as XML, JSON and text with efficiency and
effectiveness.
• The Hadoop system performs better with clusters of many servers when the
focus is on horizontal scalability.
• The system provides faster results from Big Data and from unstructured data as
well.

Yahoo has more than 1,00,000 CPUs in over 40,000 servers running Hadoop, with
its biggest Hadoop cluster running 4500 nodes as of March 2017, according to the
Apache Hadoop website.
Facebook has 2 major clusters: a cluster has 1100-machines with 8800 cores and
about 12 PB raw storage.

2.2.1 Hadoop Core Components
Core components of Hadoop

The Hadoop core components of the framework are:
1. Hadoop Common - The common module contains the libraries and utilities that are required by the
other modules of Hadoop. For example, Hadoop common provides various components and interfaces
for distributed file system and general input/output. This includes serialization, Java RPC (Remote
Procedure Call) and file-based data structures.
2. Hadoop Distributed File System (HDFS) - A Java-based distributed file system which can store all
kinds of data on the disks at the clusters.
3. MapReduce v1 - Software programming model in Hadoop 1 was using Mapper and Reducer. The v1
processes large sets of data in parallel and in batches.
4. YARN - Software for managing resources for computing. The user application tasks or sub-tasks run in
parallel at the Hadoop, uses scheduling and handles the requests for the resources in distributed
running of the tasks.
5. MapReduce v2 - Hadoop 2 YARN-based system for parallel processing of large datasets and
distributed processing of the application tasks.

2.2.1.1 Spark
• Spark is an open-source cluster-computing framework of Apache Software
Foundation.
• Spark provisions for in-memory analytics. Therefore, it also enables OLAP
and real-time processing.
• Spark does faster processing of Big Data.
• Spark has been adopted by large organizations, such as Amazon, eBay and
Yahoo.
• Several organizations run Spark on clusters with thousands of nodes.
• Spark is now increasingly becoming popular.

2.2.2 Features of Hadoop
1. Fault-efficient scalable, flexible and modular design:
- uses simple and modular programming model.
- The system provides servers at high scalability. The system is scalable by
adding new nodes to handle larger data.
- Hadoop proves very helpful in storing, managing, processing and
analyzing Big Data.
- Modular functions make the system flexible.
- One can add or replace components at ease.
- Modularity allows replacing its components for a different software tool.

2. Robust design of HDFS:
- Execution of Big Data applications continue even when an individual server
or cluster fails.
- This is because of Hadoop provisions for backup (due to replications at
least three times for each data block) and a data recovery mechanism.
- HDFS thus has high reliability.
3. Store and process Big Data:
- Processes Big Data of 3V characteristics.

4. Distributed clusters computing model with data locality:
- Processes Big Data at high speed as the application tasks and sub-tasks
submit to the DataNodes.
- One can achieve more computing power by increasing the number of
computing nodes.
- The processing splits across multiple DataNodes (servers), and thus fast
processing and aggregated results.

5. Hardware fault-tolerant:
- A fault does not affect data and application processing. If a node goes down, the
other nodes take care of the residue.
- This is due to multiple copies of all data blocks which replicate automatically.
- Default is three copies of data blocks.
6. Open-source framework:
- Open source access and cloud services enable large data store. Hadoop uses a
cluster of multiple inexpensive servers or the cloud.
7. Java and Linux based:
Hadoop uses Java interfaces. Hadoop base is Linux but has its own set of shell
commands support.

2.2.3 Hadoop Ecosystem Components

• Hadoop ecosystem refers to a combination of technologies and software
components.
• Hadoop ecosystem consists of own family of applications which tie up
together with the Hadoop.
• The system components support the storage, processing, access, analysis,
governance, security and operations for Big Data.
• The system includes the application support layer and application layer
components- AVRO, ZooKeeper, Pig, Hive, Sqoop, Ambari, Chukwa, Mahout,
Spark, Flink and Flume.

The four layers in Figure 2.2 are as follows:
(i) Distributed storage layer
(ii) Resource-manager layer for job or application sub-tasks scheduling and
execution
(iii) Processing-framework layer, consisting of Mapper and Reducer for the
MapReduce process-flow
(iv) APIs at application support layer (applications such as Hive and Pig).
The codes communicate and run using MapReduce or YARN at processing
framework layer.

• AVRO enables data serialization between the layers.
• Zookeeper enables coordination among layer components.
• Client hosts run applications using Hadoop ecosystem projects, such as Pig,
Hive and Mahout.
• Most commonly, Hadoop uses Java programming. Such Hadoop programs run
on any platform with the Java virtual-machine deployment model.
• HDFS is a Java-based distributed file system that can store various kinds of data
on the computers.

2.2.4 Hadoop Streaming
• HDFS with MapReduce and YARN-based system enables parallel processing of large
datasets.
• Spark provides in-memory processing of data, thus improving the processing speed.
• Spark and Flink technologies enable in-stream processing.
• These two lead stream processing systems and are more useful for processing a large
volume of data.
• Spark includes security features. Flink is emerging as a powerful tool.
• Flink improves the overall performance as it provides single run-time for stream
processing as well as batch processing.
• Simple and flexible architecture of Flink is suitable for streaming data.

2.2.5 Hadoop Pipes
Hadoop Pipes are the C++ Pipes which interface with MapReduce.
Java native interfaces are not used in pipes.
Apache Hadoop provides an adapter layer, which processes in pipes.
A pipe means a special file which allows data streaming from one application to
another application.
The adapter layer enables running of application tasks in C++ coded MapReduce programs.
Applications which require faster numerical computations can achieve higher throughput
using C++ when used through the pipes, as compared to Java.
Pipes do not use the standard I/O when communicating with Mapper and
Reducer codes.

2.3 HADOOP DISTRIBUTED FILE SYSTEM
Big Data analytics applications are software :-
• Applications that store and process large-scale data.
• The applications analyze Big Data using massive parallel processing frameworks.
• HDFS is a core component of Hadoop.
• HDFS is designed to run on a cluster of computers and servers at cloud-based utility
services.
• HDFS stores Big Data which may range from GB to PBs.
• HDFS stores the data in a distributed manner in order to compute fast.
• The distributed data store in HDFS stores data in any format regardless of schema.

2.3.1 HDFS Data Storage
Hadoop data store stores the data at a number of clusters.
Each cluster has a number of data stores, called racks.
Each rack stores a number of Data Nodes.
Each Data Node has a large number of data blocks.
The racks distribute across a cluster.
The nodes have processing and storage capabilities.

• The data blocks replicate by default at least on three Data Nodes
in same or remote nodes.
• Data at the stores enable running the distributed applications
including analytics, data mining, OLAP using the clusters.
• A file, containing the data divides into data blocks.
• A data block default size is 64 MBs (HDFS division of files concept
is similar to Linux or virtual memory page in Intel x86 and Pentium
processors where the block size is fixed and is of 4 KB).

Hadoop HDFS features are as follows:
(i) Create, append, delete, rename and attribute modification functions
(ii) Content of individual file cannot be modified or replaced but appended with
new data at the end of the file
(iii) Write once but read many times during usages and processing
(iv) Average file size can be more than 500 MB.

BIg Data Analytics-Module-2 vtu engineering.pptx

2.3.1.1 Hadoop Physical Organization
Conventional File System:
• The conventional file system uses directories.
• A directory consists of files.
HDFS:
• HDFS use the NameNodes and DataNodes.
• A NameNode stores the file's meta data.
• Meta data gives information about the file [example location], but NameNodes does
not participate in the computations.
• The DataNode stores the actual data files in the data blocks and participate in
computations

NameNodes and DataNodes
NameNodes:
Very Few nodes in a Hadoop cluster act as NameNodes.
These nodes are termed as MasterNodes (MN) or simply masters.
The masters have high DRAM and processing power.
The masters have much less local storage.
The NameNode stores all the file system related information such as:
• The file section is stored in which part of the cluster
• Last access time for the files
• User permissions like which user has access to the file.

NameNodes and DataNodes
DataNodes:
• Majority of the nodes in Hadoop cluster act as DataNodes and TaskTrackers.
• These nodes are referred to as slave nodes or slaves.
• The slaves have lots of disk storage and moderate amounts of processing
capabilities and DRAM.
• Slaves are responsible to store the data and process the computation tasks
submitted by the clients.

Shows the client, master NameNode, primary and secondary
MasterNodes and slave nodes in the Hadoop physical
architecture

JobTracker vs TaskTracker
JobTracker:
JobTracker which can run on the NameNode allocates the job to tasktrackers.
It tracks resource availability and task life cycle management, tracks progress of
execution and fault tolerance etc.
TaskTracker:
TaskTracker run on DataNodes.
TaskTracker runs the tasks and report the status of task to JobTracker.
It follows the orders of the job tracker and update the job tracker with its
progress status periodically.

Primary NameNode and Secondary
NameNode
• Primary NameNode is the one which stores the information of HDFS filesystem in
a file called FSimage.
• Any changes that you make in your HDFS are never written directly into
FSimage. instead, they are written into a separate temporary file.
• This node which stores this temporary file/intermediate data is called Secondary
name node.
• The Secondary namenode is sometimes used as a backup of namenode.
• The Secondary namenode collects checkpoints of metadata in namenode
and then use it in case of namenode failure.

Apache ZooKeeper
Apache ZooKeeper is a service used by a cluster (group of nodes) to coordinate
between themselves and maintain shared data with robust synchronization techniques.
ZooKeeper is itself a distributed application and provides services for writing a distributed
applications.
Distributed applications offer a lot of benefits, but they throw a few complex and
hard-to-crack challenges as well.
ZooKeeper framework provides a complete mechanism to overcome all the challenges.
- Race condition and deadlock are handled using fail-safe synchronization
approach.
Another main drawback is inconsistency of data
- which ZooKeeper resolves with atomicity.

The common services provided by ZooKeeper are as follows −
• Naming service − Identifying the nodes in a cluster by name. It is similar to DNS, but for nodes.
• Configuration management − Latest and up-to-date configuration information of the system for
a joining node.
• Cluster management − Joining / leaving of a node in a cluster and node status at real time.
• Leader election − Electing a node as leader for coordination purpose.
• Locking and synchronization service − Locking the data while modifying it. This mechanism
helps you in automatic fail recovery while connecting other distributed applications like Apache
HBase.
• Highly reliable data registry − Availability of data even when one or a few nodes are down.

Benefits of ZooKeeper
• Simple distributed coordination process
• Synchronization − Mutual exclusion and co-operation between server
processes.
• Ordered Messages
• Serialization − Encode the data according to specific rules. Ensure your
application runs consistently.
• Reliability
• Atomicity − Data transfer either succeed or fail completely, but no transaction is
partial.

2.3.1.2 Hadoop 2
Hadoop 1
• is a Master-Slave architecture. [Suffers from single point of failure]
• It consists of a single master and multiple slaves.
• Suppose if master node got crashed then irrespective of your best slave nodes,
your cluster will be destroyed.
• Again for creating that cluster you need to copy system files, image files, etc.
on another system is too much time consuming which will not be tolerated by
organizations in today’s time.

2.3.1.2 Hadoop 2
Hadoop 2
is also a Master-Slave architecture.
But this consists of multiple masters (i.e active NameNodes and standby
NameNodes) and multiple slaves.
If here master node got crashed then standby master node will take over it.
Thus Hadoop 2 eliminates the problem of a single point of failure.

2.3.1.2 Hadoop 2
Each MasterNode [MN] has the following components:
- An associated NameNode
- Zookeeper coordination (an associated NameNode), functions as a
centralized repository for distributed applications. Zookeeper uses
synchronization, serialization and coordination activities. It enables
functioning of a distributed system as a single function.
- Associated JournalNode (JN). The JN keeps the records of the state
information, resources assigned, and intermediate results or execution of
application tasks. Distributed applications can write and read data from a JN.

2.3.2 HDFS Commands
• In Core components of Hadoop [Refer figure in 2.2.1 or slide no. 28].
• Hadoop common module, which contains the libraries and utilities is at the
intersection of MapReduce, HDFS and YARN [Yet Another Resource Navigator].
• The HDFS has a shell. But this shell is not POSIX confirming.
• Thus, the shell cannot interact similar to Unix or Linux.
• Commands for interacting with the files in HDFS require /bin/hdfs <args>,
Where, args stands for the command arguments.
Full set of the Hadoop shell commands can be found at Apache Software Foundation website.
copyToLocal is the command for copying a file at HDFS to the local.
-cat is command for copying to standard output (stdout).
All Hadoop commands are invoked by the bin/Hadoop script. % Hadoop fsck / -files -blocks

Examples of usages of commands
HDFS shell
commands Example of usage
- mkdir
Assume stu_filesdir is a directory of student files in Example 2.2. Then
command for creating the directory is $Hadoop h d f s - mk d i r / u s e r / s t u
f i l e s d i r creates the directory named stu files dir
-put
Assume file stuData_id96 to be copied at stu_filesdir directory in
Example 2.2. Then $Hadoop hdfs-put stuDataid96 /user/s t u _ f i l e s d i r
copies file for student of id96 into stu_filesdir directory
-ls
Assume all files to be listed. Then $hdfs hdfs dfs-ls command does
provide the listing.
- cp
Assume stuData_id96 to be copied from stu_filesdir to new students'
directory newstu filesDir. Then $Hadoop hdfs -cp stuData _id96 /user/stu
filesdir newstu filesDir copies file for student of ID 96 into stu_filesdir
directory

2.4 MAPREDUCE FRAMEWORK AND
PROGRAMMING MODEL
MapReduce is an integral part of the Hadoop physical organization.
MapReduce is a programming model for distributed computing.
MapReduce stands for Mapper and Reducer.
What is the difference between mapper and reducer?
Mapper:
Hadoop sends one record at a time to the mapper. After that, Mappers work on one record at a
time and write the intermediate data to disk.
Reducer:
The function of a reducer is to aggregates the results, using the intermediate keys produced
by the Mapper using aggregation, query or user-specified function. Reducers write the
final concise output to the HDFS file system.

• Aggregation function means the function that groups the values of multiple
rows together to result a single value of more significant meaning or
measurement.
• For example, function such as count, sum, maximum, minimum, deviation
and standard deviation.
• Querying function means a function that finds the desired values.
• For example, function for finding a best student of a class who has shown
the best performance in examination.
MapReduce allows writing applications to process reliably the huge
amounts of data, in parallel, on large clusters of servers.

Features of MapReduce framework
1. Provides automatic parallelization and distribution of computation based on
several processors
2. Processes data stored on distributed clusters of DataNodes and racks
3. Allows processing large amount of data in parallel
4. Provides scalability for usages of large number of servers
5. Provides MapReduce batch-oriented programming model in Hadoop version
1
6. Provides additional processing modes in Hadoop 2 YARN-based system

2.4.1 Hadoop MapReduce Framework
MapReduce provides two important functions.
1. The distribution of client application task or users query to various nodes
within a cluster is one function.
2. The second function is organizing and reducing [aggregating] the results
from each node into a one cohesive response/answer.
The Hadoop framework in turns manages the task of issuing jobs, job
completion, and copying intermediate data results from the cluster
DataNodes with the help of JobTracker which is a daemon process.

What a JobTraker Does When Client Submits
the Request?
A JobTracker is a Hadoop daemon (background program).
The following are the steps on the request to MapReduce:
(i) estimate the need of resources for processing that request,
(ii) analyze the states of the slave nodes,
(iii)place the mapping tasks in queue,
(iv)monitor the progress of task, and on the failure, restart the task on slots of
time available/another node.

The job execution is controlled by two
Daemon processes in MapReduce:
JobTraker:
This process coordinates all jobs running on the cluster and assigns map.
TaskTracker:
The second is a number of subordinate processes called TaskTrackers.
These processes run assigned tasks and periodically report the progress to the
JobTracker.

2.4.2 MapReduce Programming Model
MapReduce program can be written in any language including
JAVA,
C++,
Python

2.5 HADOOP YARN
Yet Another Resource Negotiator
• YARN is a resource management platform.
• It manages computer resources.
• The platform is responsible for providing the computational resources, such
as CPUs, memory, network I/O which are needed when an application executes.
• An application task has a number of sub-tasks.
• YARN manages the schedules for running of the sub-tasks. Each sub-task uses
the resources in allotted time intervals

2.5.1 Hadoop 2 Execution Model

Figure shows the YARN-based execution model.
The figure shows the YARN components—
Client, Resource Manager (RM), Node Manager (NM), Application
Master Instance (AMI) and Containers.
List of actions of YARN resource allocation and scheduling functions is as follows:
• A MasterNode has two components: (i) Job History Server and (ii)
Resource Manager(RM).
• A Client Node submits the request of an application to the RM. The RM is
the master. One RM exists per cluster. The RM keeps information of all
the slave NMs.

The RM also decides how to assign the resources. It, therefore, performs
resource management as well as scheduling.
• Multiple NMs are at a cluster. An NM creates an AM instance (AMI) and
starts up. The AMI initializes itself and registers with the RM. Multiple AMIs
can be created in an AM.
• The AMI performs role of an Application Manager, that estimates the
resources requirement for running an application program or sub-task.
The ApplMs send their requests for the necessary resources to the RM. Each
NM includes several containers for uses by the subtasks of the application.

• NM is a slave of the infrastructure. It signals whenever it initializes. All active
NMs send the controlling signal periodically to the RM signaling their presence.
• Each NM assigns a container(s) for each AMI. The container(s) assigned at
an instance may be at same NM or another NM.
• RM allots the resources to AM, and thus to ApplMs for using assigned
containers on the same or other NM for running the application subtasks in
parallel.

2.6 HADOOP ECOSYSTEM TOOLS
Functionalities of the ecosystem tools and components
ZooKeeper -
Coordination
service
Provisions high-performance coordination service for distributed running of
applications and tasks (Sections 2.3.1.2 and 2.6.1.1)
Avro— Data
serialization and
transfer utility
Provisions data serialization during data transfer between application and
processing layers (Figure 2.2 and Section 2.4.1)
Oozie
Provides a way to package and bundles multiple coordinator and workflow jobs
and manage the lifecycle of those jobs (Section 2.6.1.2)
Sqoop
(SQL-to-
Hadoop) - A
data-transfer
software
Provisions for data-transfer between data stores such as relational DBs and Hadoop
(Section 2.6.1.3)
Flume - Large
data transfer
utility
Provisions for reliable data transfer and provides for recovery in case of failure.
Transfers large amount of data in applications, such as related to social-media
messages (Section 2.6.1.4)

Ambari - A web-based
tool
Provisions, monitors, manages, and viewing of functioning of the cluster, MapReduce, Hive and Pig
APIs (Section 2.6.2)
Chukwa - A
data collection system Provisions and manages data collection system for large and distributed systems
HBase - A
structured
data store
using database
Provisions a scalable and structured database for large tables (Section 2.6.3)
Cassandra - A
database Provisions scalable and fault-tolerant database for multiple masters (Section 3.7)
Hive - A data
warehouse system
Provisions data aggregation, data-summarization, data warehouse infrastructure, ad hoc
(unstructured) querying and SQL-like scripting language for query processing using HiveQL (Sections
2.6.4, 4.4 and 4.5)
Pig - A high-level
dataflow Provisions dataflow (DF) functionality and the execution framework.
Mahout -A machine
learning software Provisions scalable machine learning and library functions for data mining and
analytics (Sections 2.6.6 and 6.9)

2.6.1.1 Zookeeper
• Zookeeper in Hadoop behaves as a centralized repository where distributed applications can write data at a node
called JournalNode and read the data out of it.
• Zookeeper uses synchronization, serialization and coordination activities. It enables functioning of a
distributed system as a single function.
ZooKeeper's main coordination services are:
1 Name service - For example, DNS service is a name service that maps a domain name to an IP address.
Similarly, name keeps a track of servers or services those are up and running, and looks up their status by name in
name service.
2 Concurrency control - Concurrent access to a shared resource may cause inconsistency of the resource. A
concurrency control algorithm accesses shared resource in the distributed system and controls concurrency.

3 Configuration management - A requirement of a distributed system is a central
configuration manager. A new joining node can pick up the up-to-date centralized
configuration from the ZooKeeper coordination service as soon as the
node joins the system.
4 Failure - Distributed systems are susceptible to the problem of node failures.
This requires implementing an automatic recovering strategy by selecting some
alternate node for processing (Using two MasterNodes with a NameNode each).

2.6.1.2 Oozie
Apache Oozie is an open-source project of Apache that schedules Hadoop jobs.
An efficient process for job handling is required.
Analysis of Big Data requires creation of multiple jobs and sub-tasks in a process.
The two basic Oozie functions are:
• Oozie workflow jobs are represented as Directed Acrylic Graphs (DAGs), specifying a sequence of actions to execute.
• Oozie coordinator jobs are recurrent Oozie workflow jobs that are triggered by time and data availability.
Oozie provisions for the following:
1. Integrates multiple jobs in a sequential manner
2. Stores and supports Hadoop jobs for MapReduce, Hive, Pig, and Sqoop
3. Runs workflow jobs based on time and data triggers
4. Manages batch coordinator for the applications
5. Manages the timely execution of tens of elementary jobs lying in thousands of workflows in a Hadoop cluster.

2.6.1.3 Sqoop
Apache Sqoop is a tool that is built for loading efficiently the voluminous amount of data between
Hadoop and external data repositories that resides on enterprise application servers or relational
databases.
Sqoop works with relational databases such as Oracle, MySQL, PostgreSQL and DB2.
Sqoop provides the mechanism to import data from external Data Stores into HDFS.
Sqoop provides command line interface to its users. Sqoop can also be accessed using Java APIs.
Sqoop exploits MapReduce framework to import and export the data, and transfers for parallel processing
of sub-tasks.
Sqoop provisions for fault tolerance. Parallel transfer of data results in parallel results and fast data
transfer.

2.6.1.4 Flume
Apache Flume provides a distributed, reliable and available service. Flume efficiently
collects, aggregates and transfers a large amount of streaming data into HDFS.
Flume enables upload of large files into Hadoop clusters.
The features of flume include robustness and fault tolerance. Flume provides data transfer
which is reliable and provides for recovery in case of failure. Flume is useful for
transferring a large amount of data in applications related to logs of network
traffic, sensor data, geo-location data, e-mails and social-media messages.
Apache Flume has the following four important components:
1. Sources which accept data from a server or an application.
2. Sinks which receive data and store it in HDFS repository or transmit the data to
another source.

3. Channels connect between sources and sink by queuing event data for
transactions.
4. Agents run the sinks and sources in Flume. The interceptors drop the data or
transfer data as it flows into the system

2.6.2 Ambari
Ambari enables an enterprise to plan, securely install, manage and maintain the
clusters in the Hadoop.
Features of Ambari and associated components are as follows:
1. Simplification of installation, configuration and management
2. Enables easy, efficient, repeatable and automated creation of clusters
3. Manages and monitors scalable clustering
4. Provides Web User Interface and REST API (Representational State Transfer
Application Programming Interface). The provision enables automation of
cluster operations.
5. Visualizes the health of clusters and critical metrics for their operations
6. Enables detection of faulty node links
7. Provides extensibility and customizability.

2.6.3 HBase
Similar to database, HBase is an Hadoop system database.
HBase was created for large tables.
HBase is an open-source, distributed, versioned and non-relational (NoSQL)
database.
Features of HBase features are:
1. Uses a partial columnar data schema on top of Hadoop and HDFS.
2. Supports a large table of billions of rows and millions of columns.
3. Supports data compression algorithms.
4. Provisions in-memory column-based data transactions.

5. Provides random, real-time read/write access to Big Data.
6. Fault tolerant storage due to automatic failure support between DataNodes
servers.
7. Similarity with Google BigTable. HBase is written in Java.

2.6.4 Hive
Apache Hive is an open-source data warehouse software.
Hive facilitates reading, writing and managing large datasets which are at
distributed Hadoop files.
Hive uses SQL.
Hive puts a partial SQL interface in front of Hadoop.
Hive design provisions for batch processing of large sets of data.
An application of Hive is for managing weblogs.
Hive does not process real-time queries

Hive supports different storage types, such as text files, sequence files
(consisting of binary key/value pairs) and RCFiles (Record Columnar
Files), ORC (optimized row columnar) and HBase.
Three major functions of Hive are
data summarization,
query and
analysis.
Hive interacts with structured data stored in HDFS with a query language known as
HQL (Hive Query Language)

2.6.5 Pig
Apache Pig is an open source, high-level language platform.
Pig was developed for analyzing large-data sets.
Pig executes queries on large datasets that are stored in HDFS using Apache
Hadoop.
The language used in Pig is known as Pig Latin.
Pig Latin language is similar to SQL query language but applies on larger datasets.
Additional features of Pig are as follows:
(i) Loads the data after applying the required filters and dumps the data in the
desired format.
(ii) Requires Java runtime environment (JRE) for executing Pig Latin programs.

(iii) Converts all the operations into map and reduce tasks. The tasks run
on Hadoop.
(iv) Allows concentrating upon the complete operation, irrespective of the
individual Mapper and Reducer functions to produce the output results.

Mahout
Mahout is a project of Apache with library of scalable machine learning
algorithms. Apache implemented Mahout on top of Hadoop.
Apache used the MapReduce paradigm.
Machine learning is mostly required to enhance the future performance of a
system based on the previous outcomes.
Mahout provides the learning tools to automate the finding of meaningful
patterns in the Big Data sets stored in the HDFS.

Mahout supports four main areas:
• Collaborative data-filtering that mines user behavior and makes product
recommendations.
• Clustering that takes data items in a particular class, and organizes them into
naturally occurring groups, such that items belonging to the same group are similar
to each other.
• Classification that means learning from existing categorizations and then assigning
the future items to the best category.
• Frequent item-set mining that analyzes items in a group and then identifies
which items usually occur together.

BIg Data Analytics-Module-2 vtu engineering.pptx

More Related Content

What's hot (20)

Similar to BIg Data Analytics-Module-2 vtu engineering.pptx (20)

Recently uploaded (20)

BIg Data Analytics-Module-2 vtu engineering.pptx