Install Apache Hadoop for Development/Production

thanachart@imcinstitute.com1
Install Apache Hadoop for
Development/Production
using Opensource Distributions
October 2016
Dr.Thanachart Numnonda
IMC Institute
thanachart@imcinstitute.com

Thanachart Numnonda, thanachart@imcinstitute.com Oct 2016Hadoop Installation
Agenda
●
Introduction
●
Part 1: Hadoop Development Installation
●
Part 1A: Cloudera Quickstart on Cloud
●
Part 1B: Hortonworks Sandbox on Azure
●
Part 2: Hadoop Production
●
Install Cloudera Express Cluster on AWS
●
Part 3: Hadoop as a Service
●
Launch EMR cluster on AWS
Hive.apache.org

This hands-on lab aims to guide
a student to install Hadoop
for development using Sandbox
and for production on Clusters

Hadoop Distribution
●
On-Premise
●
Pure Apache Hadoop
●
Cloudera
●
Hortonworks
●
MapR
●
Pivotal
●
IBM Infosphere BigInsight
●
On-Cloud (Hadoop as a Service)
●
Amazon EMR
●
Microsoft Azure HDInsight
●
Google Cloud Platform
Hive.apache.org

The Forrester Wave: Big Data Hadoop
Distributions Q1 2016

The Forrester Wave: Big Data Hadoop
Cloud Q1 2016

Part 1: Hadoop Development

Hadoop Development: Sandbox
●
Recommended
●
Cloudera Quickstart
●
Hortonworks Sandbox
●
On PC/Mac
●
Require to install VMWare Player or VirtualBox
●
On Linux Virtual Server
●
Need to install Docker Engine
●
Microsoft Azure has a Hortonworks Sandbox Image
which can easily launched.
Hive.apache.org

http://hortonworks.com/downloads/#sandbox

http://www.cloudera.com/downloads.html

Part 1A: Install Cloudera Quickstart
on Cloud Virtual Server

Hands-On
Launch a virtual server (EC2)
on Amazon Web Services
(Note: You can skip this session if you use your own
computer or another cloud service)

Virtual Server
This lab will use a EC2 virtual server to install a
Cloudera Cluster using the following features:
Ubuntu Server 14.04 LTS
One m3.xLarge 4vCPU, 15 GB memory,80 GB
SSD
Security group: default
Keypair: imchadoop

Select a EC2 service and click on Lunch Instance

Select an Amazon Machine Image (AMI) and
Ubuntu Server 14.04 LTS (PV)

Choose m3.xlarge Type virtual server

Add Storage: 80 GB

Name the instance

Select Create an existing security group > Default

Click Launch and choose imchadoop as a key pair

Review an instance and rename one instance as a
master / click Connect for an instruction to connect
to the instance

Connect to an instance from Mac/Linux

Can also view details of the instance such as Public
IP and Private IP

Connect to an instance from Windows using Putty

Connect to the instance

Hands-On: Install a Docker Engine

Docker v.s. Hypervisor

Update OS (Ubuntu)
●
Command: sudo apt-get update

Docker Installation
●
Command: sudo apt-get install docker.io

Docker commands:
●
docker images
●
docker ps
●
docker attach id
●
docker kill id
●
docker commit id
●
Exit from container
●
exit (exit & kill the running image)
●
Ctrl-P, Ctrl-Q (exit without killing the running image)
Hive.apache.org

Install Cloudera Quickstart on
Docker Container

Pull Cloudera Quickstart
$ sudo docker pull cloudera/quickstart:latest

Verify the image was successfully pulled
$ sudo docker images

Run Cloudera quickstart
$ sudo docker run --hostname=quickstart.cloudera
--privileged=true -t -i [OPTIONS] [IMAGE]
/usr/bin/docker-quickstart
Example: sudo docker run
--hostname=quickstart.cloudera --privileged=true -t -i -p
8888:8888 cloudera/quickstart /usr/bin/docker-quickstart

Finding the EC2 instance's DNS

Login to Hue
http://ec2-54-173-154-79.compute-1.amazonaws.com:8888

Option: Launch a virtual server
on Microsoft Azure for Cloudera
Sandbox
(Note: You do not need to do this if you use your
own computer or another cloud service)

Sign up for Visual Studio Dev Essential to get free
Azure credit.

Sign in to Azure Portal

Virtual Server
This lab will use an Azure virtual server to install a
Cloudera Quickstart Docker using the following
features:
DS3_V2 Standard 4 Core, 14 GB memory,28
GB SSD

Select New => Virtual Machines => Virtual Machines

On the Basics page, enter:
1. a name for the VM
2. a username for the Admin User
3. the Authentication Type set to password
4. a password
5. a resource group name

Choose DS3_v2 Standard

Setting the inbound port for Hue (8888)

Get the IP address

ssh -i ~/.ssh/id_rsa imcinstitute@104.210.146.182

Part 1B: Install Hortonworks Sandbox
on Azure

Uncompress the docker zip file
$ gunzip HDP_2.5_docker.tar.gz
$ sudo docker load < HDP_2.5_docker.tar
(This may take several minutes)
Load the Docker image

Virtual Server
This lab will use an Azure virtual server to install a
Hortonworks Sandbox using the following features:
Hortonworks Sandbox with HDP 2.4
DS3_V2 Standard 4 Core, 14 GB memory,28
GB SSD

Select New => Search => Hortonworks Sandbox

Select Create

Set configuration as default

Get a virtual server public IP

ssh username@52.163.63.141

Login to Hortonworks >> ip_address:8888

Login to Ambari >> ip_address:8080

Part 2: Hadoop Production
on Cloud Cluster

Hadoop Production: Cluster
●
Recommended
●
Cloudera Express
●
Hortonworks
●
Cloudera Express
●
Install a cluster on 4 EC2 AWS
●
Hortonworks Data Platform
●
Install a cluster on Microsoft Azure using Marketplace
Hive.apache.org

Install Cloudera Cluster
on AWS

Cloudera VM
This lab will use a EC2 virtual server on AWS to install
Cloudera.

Hands-On: Launch a virtual server
on EC2 Amazon Web Services

Virtual Server
This lab will use a EC2 virtual server to install a
Cloudera Cluster using the following features:
Four m3.xLarge 4vCPU, 15 GB memory,80 GB
SSD
Security group: default
Keypair: imchadoop

Set the number of Instance to 4

Select Create a new security group > Add Rule as
follows

Hands-On: Installing Cloudera on EC2

Download Cloudera Manager
1) Type command >wget
http://archive.cloudera.com/cm5/installer/latest/cloudera-manager-installer.bin
2) Type command > chmod u+x cloudera-manager-installer.bin
3) Type command > sudo ./cloudera-manager-installer.bin

Login to Cloudera Manager
Wait several minutes for the Cloudera Manager Server to complete its startup.
Then running web browser: http:// public-dns: 7180

Accept the User License terms

Select Cloudera Express Edition

Provide your 4 instances <private ip>
addresses in the cluster

Select install Oracle Java & Java unlimited

Define user as ubuntu & Browse the private key (imchadoop.pem) file which we
have downloaded in the previous part. Keep Passphrase as blank

Finish

Running Hue

Part 3: Hadoop as a Service

Hadoop Production: on Cloud
●
Recommended
●
Amazon EMR
●
Azure HDInsight
●
Amazon EMR
●
Install a cluster on 4 EC2 AWS
●
Azure HD Insight
●
Install a cluster on 4 Vitual Servers Microsoft Azure
Hive.apache.org

Hands-On: Launch a EMR Cluster

Select the EMR service

Select Create Cluster

Assign configuration

Choose EC2 key pair

View SSH login instruction

Thank you
www.imcinstitute.com
www.facebook.com/imcinstitute

Install Apache Hadoop for Development/Production

More Related Content

What's hot (20)

Viewers also liked (15)

Similar to Install Apache Hadoop for Development/Production (20)

More from IMC Institute (20)

Recently uploaded (20)

Install Apache Hadoop for Development/Production